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1 Model-driven development of Web applications: the AutoWeb system 90% 

pft Piero Fraternali , Paolo Paolini 

— ACM Transactions on Information Systems (TOIS) October 2000 
Volume 18 Issue 4 

This paper describes a methodology for the development of WWW applications and a 
tool environment specifically tailored for the methodology. The methodology and the 
development environment are based upon models and techniques already used in the 
hypermedia, information systems, and software engineering fields, adapted and 
blended in an original mix. The foundation of the proposal is the conceptual design of 
WWW applications, using HDM-lite, a notation for the specification of structure, nav ... 



2 Current technological impediments to business-to-consumer electronic 88% 
commerce 

Gregory Rose , Huoy Khoo , Detmar W. Straub 
Communications of the AIS June 1999 



3 Digital library technology for locating and accessing scientific data 85% 

pjj Robert E. McGrath , Joe Futrelle , Ray Plante , Damien Guillaume 

Proceedings of the fourth ACM conference on Digital libraries August 1999 

4 Information gathering in the World-Wide Web: the W3QL query 85% 
2] language and the W3QS system 

David Konopnicki , Oded Shmueli 

ACM Transact! ns n Database Systems (TODS) December 1998 
Volume 23 Issue 4 

The World Wide Web (WWW) is a fast growing global information resource. It contains 
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an enormous amount of information and provides access to a variety of services. Since 
there is no central control and very few standards of information organization or 
service offering, searching for information and services is a widely recognized problem. 
To some degree this problem is solved by "search services/' also known as "indexers," 
such as Lycos, AltaVista, Yahoo, and others. ... 

Information retrieval on the web 82% 

Mei Kobayashi , Koichi Takeda 
ACM Computing Surveys (CSUR) June 2000 
Volume 32 Issue 2 

In this paper we review studies of the growth of the Internet and technologies that are 
useful for information search and retrieval on the Web. We present data on the 
Internet from several different sources, e.g., current as well as projected number of 
users, hosts, and Web sites. Although numerical figures vary, overall trends cited by 
the sources are consistent and point to exponential growth in the past and in the 
coming decade. Hence it is not surprising that about 85% of Internet user ... 

6 The evolution of the DARWIN system 82% 

Joan D. Walton , Robert E. Filman , David J. Korsmeyer 

— Proceedings of the 2000 ACM symposium on Applied computing March 2000 

7 Writing CGI Scripts in Python 82% 

tft Michel Vanaken 

— Linux Journal February 1997 

8 Spoken dialogue technology: enabling the conversational user interface 82% 

ACM Computing Surveys (CSUR) March 2002 

— Volume 34 Issue 1 

Spoken dialogue systems allow users to interact with computer-based applications 
such as databases and expert systems by using natural spoken language. The origins 
of spoken dialogue systems can be traced back to Artificial Intelligence research in the 
1950s concerned with developing conversational interfaces. However, it is only within 
the last decade or so, with major advances in speech technology, that large-scale 
working systems have been developed and, in some cases, introduced into 
commerc ... 

9 Greenstone: a comprehensive open-source digital library software 82% 
2) system 

Ian H. Witten , Stefan J. Boddie , David Bainbridge , Rodger J. McNab 
Proceedings of the fifth ACM conference on Digital libraries June 2000 

This paper describes the Greenstone digital library software, a comprehensive, open- 
source system for the construction and presentation of information collections. 
Collections built with Greenstone offer effective full-text searching and metadata- 
based browsing facilities that are attractive and easy to use. Moreover, they are easily 
maintainable and can be augmented and rebuilt entirely automatically. The system is 
extensible: software "plugins" accommodate different document and me ... 

10 Experiences in developing a typical web/database application 80% 

J. -P. Rosen 

Proceedings f the 2003 annual internati nal c nference n Ada: the engineering 
of c rrect and reliable software f r real-time & distributed systems using ada 
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and related technol gies December 2003 

This paper describes Gesem, an application developed internally by Adalog for 
managing the registration to its training sessions. The application features a Web 
interface that uses AWS, an interface to the MySQL DBMS (over ODBC), and a local 
interface that uses GTK. The project explored various solutions, and identified a 
number of design patterns that made the development of new functionalities very 
straightforward. The experience gained in this project can be reused for any 
development in a si ... 

11 Analysis of navigation behaviour in web sites integrating multiple 80% 

12 information systems 

Bettina Berendt , Myra Spiliopoulou 

The VLDB Journal — The International Journal on Very Large Data Bases March 
2000 

Volume 9 Issue 1 

The analysis of web usage has mostly focused on sites composed of conventional static 
pages. However, huge amounts of information available in the web come from 
databases or other data collections and are presented to the users in the form of 
dynamically generated pages. The query interfaces of such sites allow the specification 
of many search criteria. Their generated results support navigation to pages of results 
combining cross-linked data from many sources. For the analysis of visitor naviga ... 

12 Posters and Short Papers: SVG for navigating digital news video 80% 

Michael G. Christel , Chang Huang 
— Proceedings of the ninth ACM international conference on Multimedia October 
2001 

Scalable Vector Graphics (SVG) is a language for describing two-dimensional graphics 
in XML, specifically vector graphic shapes, images, and text. SVG is a new World Wide 
Web Consortium (W3C) Candidate Recommendation as of November 2000, and this 
paper describes how SVG provides an ideal framework for presenting manipulate, 
interactive summarizations into a multimedia information repository. Specifically, we 
present VIBE and map SVG interfaces into a digital news video library for delivery 
thro ... 

13 WebSplitter: a unified XML framework for multi-device collaborative 80% 
3) Web browsing 

Richard Han , Veronique Perret , Mahmoud Naghshineh 

Proceedings of the 2000 ACM conference on Computer supported cooperative 

work December 2000 

WebSplitter symbolizes the union of pervasive multi-device computing and 
collaborative multi-user computing. WebSplitter provides a unified XML framework that 
enables multi-device and multi-user Web browsing. WebSplitter splits a requested Web 
page and delivers the appropriate partial view of each page to each user, or more 
accurately to each user's set of devices. Multiple users can participate in the same 
browsing session, as in traditional conferencing groupware. Depending on the acc ... 

14 Microsoft TerraServer: a spatial data warehouse 80% 

Tom Barclay , Jim Gray , Don Slutz 

ACM SIGMOD Record , Proceedings of the 2000 ACM SIGMOD international 
conference on Management of data May 2000 
Volume 29 Issue 2 

Microsoft® TerraServer stores aerial, satellite, and topographic images of the earth in 
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a SQL database available via the Internet. It is the world's largest online atlas, 
combining eight terabytes of image data from the United States Geological Survey 
(USGS) and SPIN-2. Internet browsers provide intuitive spatial and text interfaces to 
the data. Users need no special hardware, software, or knowledge to locate and 
browse imagery. This paper describes how terabytes of "Internet unfrie ... 

15 The design and implementation of an intentional naming system 80% 

Cft William Adjie-Winoto , Elliot Schwartz , Hari Balakrishnan , Jeremy Lilley 

— ACM SIGOPS Operating Systems Review , Proceedings of the seventeenth ACM 

symposium on Operating systems principles December 1999 

Volume 33 Issue 5 

This paper presents the design and implementation of the Intentional Naming System 
(INS), a resource discovery and service location system for dynamic and mobile 
networks of devices and computers. Such environments require a naming system that 
is (i) expressive, to describe and make requests based on specific properties of 
services, (ii) responsive, to track changes due to mobility and performance, (iii) 
robust, to handle failures, and (iv) easily configurable. INS uses a simple language 
based o ... 

16 Catching the boat with Strudel: experiences with a Web-site 80% 
2) management system 

Mary Fernandez , Daniela Florescu , Jaewoo Kang , Alon Levy , Dan Suciu 

ACM SIGMOD Record , Proceedings of the 1998 ACM SIGMOD international 

conference on Management of data June 1998 

Volume 27 Issue 2 

The Strudel system applies concepts from database management systems to the 
process of building Web sites. Strudel's key idea is separating the management of the 
site's data, the creation and management of the site's structure, and the visual 
presentation of the site's pages. First, the site builder creates a uniform model of all 
data available at the site. Second, the builder uses this model to declaratively define 
the Web site's structure by applying a "site-definition query" ... 

17 Using the WWW as the delivery mechanism for interactive, visualization- 80% 
2j based instructional modules (report of the ITiCSE '97 working group on 

visualization) 

Thomas Naps , Joseph Bergin , Ricardo Jimenez-Peris , Myles F. McNally , Marta Patino- 
Martmez , Viera K. Proulx , Jorma Tarhio 

The supplemental proceedings of the conference on Integrating technology into 
computer science education: working group reports and supplemental 
proceedings June 1997 

18 Using the WWW as the delivery mechanism for interactive, visulaization- 80% 
2) based instructional modules: report of the ITiCSE *97 working group on 

visualization 

Thomas Naps , Joseph Bergin , Ricardo Jimenez-Peris , Myles F. McNally , Marta Patino- 
Martmez , Viera K. Proulx , Jorma Tarhio 
ACM SIGCUE Outl ok October 1997 
Volume 25 Issue 4 

19 Visualization products on-demand through the Web 80% 

Suzana Djurcilov , Alex Pang 
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Proceedings of the third symposium on Virtual reality m deling language 

February 1998 



20 Online aggregation 

Cft Joseph M. Hellerstein , Peter J. Haas , Helen J. Wang 

L — 1 ACM SIGMOD Record , Proceedings of the 1997 ACM SIGMOD international 
conference on Management of data June 1997 
Volume 26 Issue 2 

Aggregation in traditional database systems is performed in batch mode: a query is 
submitted, the system, processes a large volume of data over a long period of time, 
and, eventually, the final answer is returned. This archaic approach is frustrating to 
users and has been abandoned in most other areas of computing. In this paper we 
propose a new online aggregation interface that permits users to both observe the 
progress of their aggregation queries and control execution on ... 
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US006181838B1 

(12) United States Patent m Patent No.: us 6,181,838 Bl 

Knowlton (45) Date of Patent: Jan. 30, 2001 



(54) MECHANISM FOR THE CAPTURE OF 
GRAPHICAL REPRESENTATIONS 

(75) Inventor: Kenneth Charles Knowlton, 

Merrimack, NH (US) 

(73) Assignee: QuickBuy, Inc., Acton, MA (US) 

( * ) Notice: Under 35 U.S.C. 154(b), the term of this 
patent shall be extended for 0 days. 

(21) Appl. No.: 09/302,345 

(22) Filed: Apr. 29, 1999 

Related U.S. Application Data 

(62) Division of application No. 08/814,118, filed on Mar. 10, 
1997. 

(51) Int. CI. 7 G06K 9/54 

(52) U.S. CI 382/305; 382/312 

(58) Field of Search 382/305, 180, 

382/199, 312, 100, 325; 707/348 

(56) References Cited 

U.S. PATENT DOCUMENTS 

5,684,999 * 11/1997 Okamoto 382/305 

5,737,440 * 4/1998 Kunkler 382/175 

5,848,186 * 12/1998 Wang ct al 382/180 



5,940,535 * 8/1999 Huang 382/199 

* cited by examiner 

Primary Examiner— -Yon J. Couso 

(74) Attorney, Agent, or Firm — G. D. Clapp, Esq. 

(57) ABSTRACT 

A visual link mechanism residing in a local system for 
identifying addresses of locations in the plurality of remote 
systems wherein the local system is connected through a 
network to the plurality of remote systems. The visual link 
mechanism includes a visual link library and a network 
access mechanism responsive to a visual link including a 
display able graphic icon for accessing the location repre- 
sented by a selected graphic icon. Various structures of 
visual links are described, each being an entity existing 
independently of the system environment in which it resides, 
and the network access mechanism includes a layout table 
for storing a plurality of plans for arranging and displaying 
a plurality of visual link graphic icons in a display, a visual 
links organizer, a visual link screen saver, and a hash 
protection mechanism for detecting the unauthorized con- 
struction or modification of visual links or other forms of 
files. Also described is a visual link capture engine for 
extracting graphics information from a data file and gener- 
ating a corresponding graphic icon and a display layout 
generator for generating display layouts of sets of predeter- 
mined numbers of displayable visual objects. 

15 Claims, 17 Drawing Sheets 
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MECHANISM FOR THE CAPTURE OF 
GRAPHICAL REPRESENTATIONS 

CROSS REFERENCES TO RELATED 

APPLICATIONS 5 

The present Patent Application is a Divisional Patent 
Application of co-pending Prior patent application Ser. No. 
08/814,118, filed Mar. 10, 1997 and which is incorporated 
herein by reference and is related to U.S. Patent Appellation 
Serial No. by Kenneth Charles Knowlton and Gary Steven 10 
Miliefsky for A PROTECTION MECHANISM FOR 
VISUAL LINK OBJECTS and to U.S. patent application 
Ser. No. 09/302344 by Kenneth Charles Knowlton and 
Gary Steven Miliefsky for A DISPLAY LAYOUT GEN- 
ERATOR FOR GRAPHICAL REPRESENTATIONS, both is 
of which are filed on even date with the present Patent 
Application. 

FIELD OF THE INVENTION 

The present invention relates to a method and apparatus 2 o 
for the graphical representation of bodies or sources of 
information and, in particular, a method and apparatus for 
capturing and displaying graphic representations of bodies 
or sources of information and for indexing and accessing the 
bodies or sources of information. 25 

BACKGROUND OF THE INVENTION 

The volume of information accessible to users of com- 
puter systems, either directly on a local system or indirectly 
through a network, such as the internet and the World Wide 30 
Web constructed on the internet, usually referred to as "the 
Web", has increased to the point where tracking and index- 
ing the information so that the user can locate a given body 
or source of information has become a primary problem. For 
example, users may have literally thousands of documents, 35 
spreadsheets, and graphics files directly available on their 
local system and as many universal resource locators 
(URLs), wherein each URL essentially points to or is an 
address of a location of a body or source of information on 
the internet and contains the information required to direct 40 
a TCP/IP based protocol executing on a system to a par- 
ticular location or resource on the internet and on intranets. 

Hie problem is compounded in that the methods and 
means in general use for indexing such information remains 
relatively primitive compared to the methods and means for 45 
generating and accessing the information. 

For example, it is well known and widely accepted that 
humans comprehend complex or voluminous information 
more readily when it is presented in graphical form rather 
than as text or numbers. This characteristic is, in fact, widely 50 
exploited in computer operating environments such as 
Microsoft Windows and Apple Macintosh and in most 
applications programs, such as Netscape Navigator, and 
even in much of the information itself, which is frequently 
graphical in nature or uses or incorporates graphical infor- 55 
mation. 

The methods presently in use for indexing information or 
sources of information, however, generally represent bodies 
or sources of information, such as files or URLs, in text 
form, that is, as lists of alphanumeric names or designators. 60 
For example, even the computer operating environments and 
applications programs providing graphical user interfaces, 
such as Microsoft Windows, the Apple Macintosh and 
Netscape Navigator, either use alphanumeric designators 
alone or use alphanumeric designators to distinguish 65 
between otherwise identical icons representing files generi- 
cally. 
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The systems of the prior art have generally retained 
alphanumeric designators to represent bodies or sources of 
information because of the difficulties inherent in represent- 
ing each individual file or source of information with a 
graphic representation that is sufficiently distinguishable 
from other graphic representations of files or sources of 
information and that presents a visually strong image to a 
user. 

The primary problem in this respect is the volume of data 
that must be stored to represent each file or source of 
information individually when there are potentially hun- 
dreds of such files or sources of information to individually 
designate. That is, it has been generally accepted in the prior 
art that it is necessary to have a relatively large and complex 
graphic representation of each file or source of information 
in order to have visually strong and distinguishable repre- 
sentations of files or sources of information when there is 
more than a relatively limited number of files or sources of 
information. As such, only a few specialized applications 
programs, such as the Corel graphics programs and, for 
example, hypertext markup language (HTML) pages, such 
as frequently used on the Web, have attempted to provide 
any form of graphically based indexing of files. These 
programs and HTML pages have generally used compressed 
versions of full size images for this purpose, which are in 
themselves large graphics files, thereby requiring long 
extensive storage space and long transmission times to 
transfer over a network. 

Related problems are, of course, the overwhelming effort 
necessary to generate what may be literally hundreds of 
unique representations to be associated with various files 
and sources of data, and the inherent problems in organizing 
and displaying such graphic representations for files and 
information sources on a typical display. 

Still other related problems are, for example, the detection 
of the unauthorized construction or modification of com- 
puter files, such as visual links, and an effective method for 
providing encrypted authorization within a file, again such 
as a visual link, without increasing the size of the file or 
causing difficulties for human and system readers of the file. 

The present invention provides a solution to these and 
other problems of the prior art. 

SUMMARY OF THE INVENTION 

The present invention is directed to a visual link mecha- 
nism residing in a local system for identifying addresses of 
locations in the plurality of remote systems wherein the local 
system is connected through a network to a plurality of 
remote systems and typically includes a memory for storing 
data and programs, a processor operating under control of 
the programs to perform operations on the data, and a 
display. 

According to the present invention, the visual link mecha- 
nism includes a visual link library for storing visual links 
wherein each visual link corresponds to an address of a 
location in the plurality of remote systems, and a network 
access mechanism connected to the network for displaying 
the visual links to a user and allowing the user to select a 
visual link representing a corresponding location in a remote 
system. The network access mechanism is responsive to the 
visual link selected by the user for accessing the location 
represented by the selected visual link, and each visual link 
includes a graphic icon providing a displayable image 
representing the corresponding location and the address of 
the corresponding location. 

Each visual link may also include alphanumeric informa- 
tion relating to the corresponding location, including a title 
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identifying the location and text describing the location, and 
may include keywords relating to the contents of the loca- 
tion. 

Each visual link preferably may include the graphic icon 
providing a display able image representing the correspond- 5 
ing location and a visual link data set containing information 
relating to the location, including the address of the corre- 
sponding location, wherein each visual link dataset may 
include a plurality of designators, each designator storing 
corresponding information relating to the corresponding 10 
location and including a graphic icon designator for storing 
an identification of the corresponding graphic icon, an 
address designator for storing the address of the correspond- 
ing location, and an end designator for indicating the end of 
the visual link dataset. 15 

The designators may further include a subject designator 
for storing text information identifying the contents of the 
location, an author designator for storing text information 
identifying the author of the contents of the location, and a 
keyword designator for storing key words relating to the 2 o 
contents of the location. 

According to the present invention, the visual link mecha- 
nism further includes a link for relating the graphic icon and 
the visual link dataset. In one implementation, the link 
comprises a file for storing a pathname of the graphic icon 25 
and a pathname of the visual link dataset and in another the 
link comprises a graphic icon pointer designator stored in 
the visual link dataset for storing an identification of the 
graphic icon. In still other implementations, the graphic icon 
and the visual link dataset are stored in a single file, the 30 
graphic icon being stored starting at the beginning of the file 
and the visual link dataset following the graphic icon, 
wherein the visual link dataset further includes a length of 
dataset number stored at the end of the file and indicating the 
location of the start of the visual link dataset relative to the 35 
end of the file. In yet other implementations, the visual link 
dataset further includes an address pointer and an end 
pointer stored preceding the length of dataset number and 
indicating the locations of the address designator and the end 
designator in the file. 40 

Further according to the present invention, the network 
access mechanism includes a layout table for storing a 
plurality of plans for arranging and displaying a plurality of 
graphic icons in display spaces wherein each plan corre- 
sponds to a display space of predetermined dimensions and 45 
to a predetermined number of graphic icons of predeter- 
mined dimensions and defining the arrangement of the 
graphic icons in the display space, a input for providing a set 
of display definition inputs wherein each set of display 
definition inputs includes a display space definition identi- 50 
fying the dimensions of a display space and a set of visual 
object definitions identifying a set of graphic icons of a 
predetermined number and of predetermined dimensions to 
be displayed in the display space. A display generator is 
responsive to the display definition inputs for reading from 55 
the layout memory a plan corresponding to a set of display 
definition inputs and generating a display of the identified 
set of graphic icons in the display space according to the 
corresponding plan and displaying the identified set of 
graphic icons on the display. In one implementation, the 60 
input for providing a set of display definition inputs includes 
a visual link display memory for storing groups of identi- 
fications of visual links, each group of identifications of 
visual links being a set of visual object definitions identi- 
fying a set of graphic icons of a predetermined number and 65 
of predetermined dimensions to be displayed in the display 
space. 
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Still further according to the present invention, the net- 
work access mechanism includes a visual links organizer 
connected to the display for displaying the visual links to a 
user and allowing the user to select a visual link representing 
a corresponding location in a remote system and a network 
communication mechanism connected to the visual links 
organizer and responsive to the visual link selected by the 
user for accessing the location represented by the selected 
visual link. 

The visual link mechanism of the present invention may 
also include a visual link screen saver connected from the 
visual link library and to the display and responsive to 
operations of the local system for generating a visual link 
screen saver display of selected graphic icons on the display 
wherein the screen saver is responsive to an indication of a 
displayed graphic icon of the visual link screen saver display 
for invoking the a network access mechanism for accessing 
the location represented by the selected visual link corre- 
sponding to the selected graphic icon. 

In other aspects, the present invention includes a hash 
protection mechanism for detecting the unauthorized con- 
struction or modification of visual links or other forms of 
files, including a file reader for reading an encryption part of 
a file and a stored hash encryption value stored in a storing 
part of the file, a hashing mechanism for generating an 
encrypted hash value from the data contained in the encryp- 
tion part of the file, and a comparator for comparing the 
generated encrypted hash value and the stored hash value 
and determining when the generated and stored hash values 
are equal. According to the present invention, the storing 
part of the file includes data of the file separate from the 
encrypting part of the data of the file and the data of the 
storing part of the file is characterized in having alternate 
and semantically equivalent expressions wherein the stored 
hash encryption value is encoded in the data of the storing 
part of the file using the alternate and semantically equiva- 
lent expressions representing the data of the storing part of 
the file to encode the stored hash encryption value. 

The present invention also includes a capture engine for 
extracting graphics information from a data file and gener- 
ating a corresponding graphic icon forming a displayable 
image representing the graphics information. According to 
the present invention, the capture engine includes a gray- 
scale image generator for receiving an original image and 
generating a corresponding grayscale image containing 
brightness values representing the original image, an edge 
image mechanism for receiving the grayscale image and 
generating a corresponding edge image representing areas of 
visually significant graphic structure in the grayscale image 
as represented by areas of change in the brightness values, 
a candidate region search mechanism for receiving the edge 
image and identifying initial candidate regions of the edge 
image representing visually significant areas of the original 
image, and a candidate region adjustment and comparator 
mechanism for selecting a candidate region to be used in 
generating a corresponding graphic icon and for adjusting 
the selected candidate region to conform to predetermined 
dimensions for a graphic icon. 

In further aspects of the present invention, the grayscale 
image generator includes a grayscale lookup table for stor- 
ing grayscale brightness values corresponding to each of the 
possible brightness values of image elements in original 
images, a grayscale converter for reading the brightness 
values of the image elements of the original image, reading 
the corresponding grayscale brightness values stored in the 
grayscale lookup table, and generating a corresponding 
grayscale image wherein each image element of the original 
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image is represented by a grayscale image element having the output edge image and the output edge image element 

the grayscale brightness value corresponding to the bright- corresponding to the sum element. A rectangular sum search 

ness value of the image element. mechanism is connected from the rectangular sum array 

A1 . 4 . 4 . „ u * generator to receive the rectangular sum array for examining 

Also according to the present invention, each image f L vuvl . , , & . , . uf.u 

. A ■ • , • • 4 jl * r i i the summed edge values represented in each or the sum 

element of the original image is represented by a set of color * «v ^uiuivu 6 y 

, ° . f / • 1 ♦ r ,u elements of the rectangular sum array and identifying at 

values the set of color values of an image element of the ^ om ^ region having a high summed 

original element representee the color and brightness value cd ygl ^ ^ rcgion u^by rep resent- 

of the image element. Accordingly, the grayscale lookup e ^ c ondi to a ion of the ori inal ^ 

table contains a grayscale brightness value corresponding to ^ a ^J^, visual structure 

each of the possible sets of color values of image elements 10 „ .,, , ~f .. , , ... , . 

in original images and the grayscale converter is responsive S ,U , the candldale re Si° n a ° d «»• 

to tie set of lolor values of each image element of the P» rat0 < m f hamsm . 7 clud « a , D ed S. e «*f er for ( recel . v,n S 

. . , . c 4 . ° i- the at least one lmtial candidate region and generating from 

original image for reading the corresponding grayscale ...... j-j. • j- j- 

brightness values stored in the grayscale lookup table and inmal " ndlda e K f° n » corresponding grown candi- 

generating a corresponding grayscale image wherein each « dateregtonby adjustmg the extenu of each imUa candidate 

. , . ff u . * k„ o region to include all portions of significant visual structures 

image element of the original image is represented by a b . f . , , . . . . , 

grayscale image element having the grayscale brightness havin g a P art thereo f f contam « 1 w,hm the ' D \ tial candlda ' 6 

value corresponding to the set of color values of the image K &°»' gyrating from each grown candidate region a 

element corresponding edge adjusted candidate region by adjusting 

20 tne extents of each grown candidate region so that each edge 

Also, the grayscale image generator generates each image adjusted candidate region conforms t0 one of a set of 

element of the grayscale image from a plurality of corre- predetermined exten ts for graphic icons, comparing each 

spending elements of the original image so that the number edge ad j usted cand idate region with others of the edge 

of image elements in the grayscale image is proportionally adjustcd candidatc rcgions md selecting the edge adjusted 

reduced from the number of image elements m the corre- ^ candidate region jiving lhe highest amoiint of significant 

sponding original image. visual structure> and 

comparing each selected edge adjusted 

According to the present invention, the edge image candidate region with previously existing graphic icons, 

mechanism includes an edge image generator for receiving When the selected edge adjusted candidate region is similar 

'an input grayscale image and generating a corresponding ^ a previously existing graphic icon, the edge adjuster 

output edge image wherein each image element of an output 3Q se lects a next one of the at least one edge adjusted candidate 

edge image is represented by an edge value proportionate to regions, and, when a selected candidate region is distin- 

the difference between the brightness value of the corre- guished from the previously existing graphic icons, identi- 

sponding image element in the input grayscale image and fi es tne corresponding region of the original image for use as 

the brightness values of the neighboring image elements of a gra phic icon. 

the corresponding image element in the input grayscale 3f . ^ present i nvent ion also provides a display layout 

image. An output edge image thereby represents areas of generator for generating layouts of sets of predetermined 

visually significant graphic structure in the input grayscale num bers of displayable visual objects of predetermined 

image as represented by areas of change in the brightness dimensions into a display area of predetermined dimensions, 

values of the input grayscale image. According to the present invention, the display layout gen- 

The edge image mechanism may also include a lowpass 40 era t or includes a possible layouts generator for receiving a 

filter connected between the grayscale image generator and display space definition defining the dimensions of the 

the edge image generator for receiving a grayscale image display area and a visual object definition defining the 

and generating a corresponding filtered grayscale image of numbers and dimensions of the visual objects to be arranged 

reduced resolution and provides the filtered grayscale image m the display area and generating all possible layouts of the 

to the edge image generator as the correspond input gray- 45 visual objects in the display area by placing in a layout a 

scale image to the edge image generator, wherein each visual object and searching forward with a next visual object 

image element of a filtered grayscale image is determined as to identify a placement of the next visual object and back- 

a proportioned summation of the brightness values of tracking on each next placed visual object by removing the 

selected neighboring image elements of the image element ne xt placed visual object and selecting and placing a differ- 

of the grayscale image so that a filtered grayscale image 50 ent next visual object until all combinations of visual objects 

thereby represents visual structures of significant extents of m $ placements of visual objects have been exhausted, 

the grayscale image. The d jspi a y layout generator further includes a layout 

In further aspects of the present invention, the edge image memory for storing each arrangement of the set of visual 
generator generates each image element of an output edge objects in the display area generated by the possible layouts 
image from a plurality of corresponding elements of the 55 generator, a layout test mechanism for testing each new 
input grayscale image so that the number of image elements layout by comparing each new layout with previously exist- 
in the output edge image is proportionally reduced from the ing layouts to determine when the new layout has a sub- 
number of image elements in the corresponding grayscale population of a previously existing layout or matches a 
input image. previously existing layout and when the previously existing 

Still further according to the present invention, the can- 60 layout has a subpopulation of the new layout, wherein a 

did ate region search mechanism includes a rectangular sum subpopulation is a subset of the visual objects, a layout 

array generator for receiving an output edge image and previous/next selector for selecting the new layout or the 

generating a corresponding rectangular sum array having a previously existing layout by discarding the new layout 

sum element corresponding to each image element of the when the new layout has a subpopulation of a previously 

output edge image wherein each sum element represents the 65 existing layout or matches a previously existing layout and 

sum of the edge values of the image elements of a region of replacing the previously existing layout with the new layout 

the output edge image bounded by a coordinate location of when the previously existing layout has a subpopulation of 
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the new layout, a layouts comparator for comparing pairs of 
layouts stored in the layout generator memory and discard- 
ing any layout that contains a subpopulation of another 
layout, and a layout table for storing the final layouts for use 
in generating visual displays of the sets of visual objects in 5 
the display areas. 

According to the present invention, the layout generator 
may further include a screen map generator for receiving the 
display space definition and the visual object definition and 
generating a screen map corresponding to the defined dis- 10 
play area and having instance locations representing pos- 
sible placement locations of the visual objects in the display 
area, an instance pattern generator for receiving the screen 
map and the visual object definition and generating, for each 
type of visual object of the set of visual objects wherein a 15 
type of visual object is defined by the dimensions of a visual 
object, at least one instance pattern representing all possible 
instances of locations of the visual object in the screen map, 
and an instance pattern enumerator for enumerating the 
resulting instance patterns and providing the enumerated 20 
instance patterns to the possible layouts generator to be used 
as initial starting points in generating layouts. 

The layout generator may further include a screen spatial 
division generator connected from the possible layouts gen- 
erator for receiving the display space definition and the 25 
enumerated instances patterns and dividing the defined 
display area into spatial divisions along axes of symmetry, 
a location instances sorter for sorting, for each spatial 
division of the display area, the enumerated instance pat- 
terns of visual objects in each spatial division by visual 30 
object type, and generating a set of location instances of 
visual objects for each spatial division, a location instance 
search and compose mechanism for searching for and 
generating, from the sets of location instances of visual 
objects for each spatial division, at least one combination of 35 
location instances in at least one spatial division wherein 
each combination of location instances contains a combina- 
tion of visual objects of the set of visual objects, and a 
super-object generator for designating each combination of 
visual objects as a visual super-object and providing the 40 
designations to the possible layouts generator as respresent- 
ing corresponding visual object. 

The layout generator may also include a comer area 
examiner connected from the possible layouts generator for 45 
receiving the layouts and examining the comer areas of the 
layouts for unique combinations of visual objects and dis- 
carding layouts having comer areas having layouts of visual 
objects that are rotational or reflectional equivalents of a 
previously existing layout. 5Q 

Finally, the layout generator may also include a screen 
sub-area divider connected from the possible layouts gen- 
erator for receiving the display area definition and dividing 
the display area into sub- areas and provide to the possible 
layouts generator a display space definition for each sub- 55 
area for use in generating layouts for each sub-area and a 
screen sub -area combiner connected from the layout com- 
parator for composing layouts for the display area by 
combining the layouts for the sub -are as and provide the 
combined layouts to the layout comparator as the layout for 60 
the defined display area. 

DESCRIPTION OF THE DRAWINGS 

The foregoing and other objects, features and advantages 
of the present invention will be apparent from the following 65 
description of the invention and embodiments thereof, as 
illustrated in the accompanying figures, wherein: 
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FIG. 1A is a block diagram of a system embodying the 
present invention; 

FIG. IB is diagrammatic representations of a database 
entry and a visual link; 

FIG. 1C illustrates the embedding of a visual link refer- 
ence in a hypertext document; 

FIG. ID illustrates the designation of an area of a hyper- 
text document to comprise a graphic icon; 

FIG. 2A is a diagrammatic representation of the image 
data structures generated and used in capturing a graphic 
icon; 

FIG. 2B illustrates the arrangement or tiling of graphic 
icons with borders in a display; 

FIGS. 3A-1 and 3A-2 are block diagrams of a capture 
engine of the present invention; 

FIGS. 3B, 3C, 3D, 3E and 3F are diagrammatic repre- 
sentations of scaling and edge determination performed by 
the capture engine of the present invention; 

FIGS. 3G and 3H illustrate the use of a rectangular sum 
array to determine area content by the capture engine of the 
present invention; 

FIG. 4A is a block diagram of a visual links organizer for 
generating display layouts and plans for graphic icons; 

FIGS. 4B, 4C and 4D are illustrations of graphic icon 
layouts and plans; 

FIGS. 4E-1 through 4E-5 are block diagrams illustrating 
the generation of graphic icon layouts and plans by the 
visual links organizer; 

FIGS. 4F, 4G, 4H, 41, 4J, 4K, 4L, and 4M are illustrations 
of data structures and layouts used in the operation of the 
visual links organizer in generating graphic icon layouts and 
plans; 

FIGS. 5A through 5G illustrate embodiments of visual 
links according to the present invention; and, 

FIGS. 6A and 6B illustrate a hashing protection mecha- 
nism for detecting the unauthorized construction or modi- 
fication of a file such as a visual link. 

DESCRIPTION OF THE INVENTION 

As described above, the present invention is a system and 
a method and apparatus for the graphical representation of 
bodies or sources of information, such as files or internet or 
Web locations, and, in particular, for capturing and display- 
ing graphic representations of the bodies or sources of 
information for use in indexing and accessing the bodies or 
sources of information. The following will first describe a 
presently preferred embodiment of a system for indexing 
and displaying and accessing information on the World Wide 
Web, hereafter referred to as the Web, and will then describe 
components of that system for capturing and displaying 
graphic representations of bodies of information, such as 
files, or sources of information, such as sites or HTML 
documents, on the Web and the use of such graphical 
representations to access such bodies or sources of infor- 
mation. It will be appreciated by those of ordinary skill in the 
arts, however, that the present system for indexing and 
displaying and accessing information accessible across or 
through a network is not limited to the World Wide Web and 
HTML documents residing therein, but is also applicable for 
indexing and displaying and accessing information acces- 
sible across or through other forms of networks, such as 
electronic mail systems, wherein a location on such a 
network is identified by some form of pointer or address and 
wherein locations, files or persons on such a network are 
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visually identifiable to a user, for example, by means of a 
representative graphic icon or, in an electronic mail system, 
a photograph of a recipient. The following descriptions, 
however, will focus on exemplary embodiments of the 
present invention as implemented for use on World Wide 5 
Web systems as representative of all generally similar types 
of networks wherein users wish to identify, index and access 
locations, files or persons on the networks. 

Finally, it will be noted that in the following each refer- 
ence number is a three or four digit number comprised of 10 
two rightmost digits and one or two leftmost digits wherein 
the one or two leftmost digits refer to the number of the 
figure in which the referred to element first appears while the 
right two digits identify the element within the figure in 
which the element first appears. Once assigned, a given 15 
reference number is used throughout the subsequent figures 
and descriptions to refer to the corresponding element. 

It will also be noted that closely related figures, that is, 
figures illustrating related discussions in the following 
description, are indicated as a group of related figures by 20 
assigning a figure number to the group of figures and 
distinguishing among the figures of a group by alphabetic 
suffixes, such as FIG. 1A, FIG. IB, and so on. In addition, 
certain figures are contained on multiple drawings and, in 
such instances, the individual drawings comprising the 25 
figures and the portions of the figures appearing thereupon 
are indicated and referred to by a figure number with an 
appended drawing number; for example, a FIG. 3Amay be 
related to FIGS. 3B and 3C and may be comprised of three 
drawings identified as FIG. 3A-1, FIG. 3A-2 and FIG. 3A-3. 30 
In a similar manner, certain figures may be optionally 
combined with other figures, for example, to illustrate 
alternate or expanded embodiments of the subject matter 
illustrated in a first figure, and in these instances the figures 
are again identified by a figure number with an appended 35 
drawing number, so that a mechanism illustrated in a FIG. 
4E-1 may be extended or expanded by additional compo- 
nents or mechanisms illustrated in FIGS. 4E-2, 4E-3, and so 
on. 

40 

A. DESCRIPTION OF A PREFERRED 
EMBODIMENT OF THE INVENTION 

1 . Description of a System 110 for Capturing and 
Displaying Bodies and Sources of Information, and 45 
Accessing the Information (FIGS. 1A, IB, 1C and 
ID) 

Referring to FIG. 1A, therein is shown a block diagram of 
a System 110 incorporating the present invention. The 50 
System 110 is illustrated therein as having a Local System 
112 connected to a Server System 114 through a Network 
116. In the present discussion, and for purposes of illustra- 
tion of an embodiment of the present invention, Network 
116 is the internet and, in particular, the World Wide Web 55 
(Web) implemented on the internet, and Server System 114 
is a World Wide Web server while Local System 112 and 
Server System 114 communicate with one another and with 
other Web Servers 118 through an appropriate communica- 
tions protocol, such as TCP/IP. 60 

Referring first to Server System 114, as shown in FIG, 1A 
Server System 114 includes a Network Service System 120 
for communicating with Local System 112 and Web Servers 
118 and a Network Data Collector and Indexing Server 122, 
such as a Web "Spider", for automatically searching Net- 65 
work 116 to locate Web Servers 118 and to access, capture 
and retrieve information stored on Web Servers 118, such as 
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in HTML documents or other forms of files. As is well 
known in the relevant arts, Network Data Collector and 
Indexing Server 122 operates to construct an Indexed Data- 
base 124 containing the information retrieved from Web 
Servers 118 wherein the retrieved information, is repre- 
sented in FIG. 1A as stored in Database Entries 126. 

As illustrated in FIG. IB, each Database Entry 126 will 
typically alphanumeric information that include a Universal 
Resource Locator (URL) 128, often referred to as a "fink", 
identifying the location or address on the Web of a corre- 
sponding HTML document, a Title 130 containing text of 
any arbitrary length and usually comprising a name of the 
HTML document, a Text 132 of any arbitrary length and 
usually comprising a brief description of the HTML docu- 
ment or its contents. Each Database Entry 126 may also 
include a Date 134 containing the date and time the "link" 
was last updated, and may include Keywords 136, which 
may contain one or more keywords characterizing the 
HTML document and used to search for and locate the 
corresponding Database Entry 126. In other 
implementations, such as electronic mail systems, for 
example, the role of URLs in the following discussions may 
be fulfilled by electronic mail addresses of various forms, in 
a manner that will be well and easily understood by those or 
ordinary skill in the relevant arts. 

As is well known and understood in the relevant arts, a 
user of Local System 112 may use a web program executing 
on Local System 112, such as Netscape Navigator, to access 
the Web and, with particular regard to the present invention, 
to access Server System 114 to search for Web Servers 118 
having information sought by the user. Server 114 will then 
access and search Indexed Database 124 to locate Database 
Entries 126 relevant to the user's request, and will return at 
least portions of the Database Entries 126 to Local System 
112, such as the URL 128, Title 130 and Text 132. The user 
of Local System 112 may then use the returned URL's 128 
to access and retrieve the corresponding HTML documents 
from Web Servers 118 and may store a URL 128 returned 
from Server System 114 for subsequent use, together with its 
associated information. The user of Local System 112 may 
also, as is common practice, capture and store URLs 128 of 
documents and sites and their associated information, such 
as Title 130, and perhaps Text 132, from a document or site 
accessed on a Web Server 118. Again, in other 
implementations, such as electronic mail systems, for 
example, the role of a web server and index may be fulfilled 
by electronic mail servers of various forms and electronic 
mail indexes residing on electronic mail servers, in a manner 
that will be well and easily understood by those or ordinary 
skill in the relevant arts. 

According to the present invention, Server System 114 
further includes a Visual Links Automatic Capture Engine 
138 A which, as described in further detail in the following, 
accesses HTML pages located in Web Servers 118 by 
Network Data Collector and Indexing Server 122 and con- 
structs a Visual Links Database 140 which contains Visual 
Links 142 corresponding to the HTML documents indexed 
in Database Entries 126. As shown in FIG. IB, and as will 
be discussed in further detail in a subsequent description, a 
Visual Link 142 will include alphanumeric information 
including a Universal Resource Locator (URL) 128 and may 
include a Title 130, a Text 132, and a Date 134, and may 
include Keywords 136. 

In addition, however, a Visual Link 142 includes a 
Graphic Icon 144 which is generated or extracted from the 
graphics and text information present in the corresponding 
HTML document, or other type of file, by Visual Links 
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Automatic Capture Engine 138A and which, as described in 
further detail in the following, serves as a visual represen- 
tation to the user of the associated document or file. Yet 
again, in other implementations, such as electronic mail 
systems, for example, the role of Graphic Icons 144 in the 
following discussions may be fulfilled by graphic represen- 
tations of various forms of files, locations or recipients on a 
network, such as photographs of electronic mail recipients, 
in a manner that will be well and easily understood by those 
or ordinary skill in the relevant arts. 

In alternate embodiments of Server System 114, for 
example, one which operates entirely with Visual Links 142, 
Visual Links Database 140 may entirely replace Indexed 
Database 124. In more typical embodiments, for example, 
until or unless Visual Links 142 are universally adopted, 
Server System 116 will operate with a mixture of text only 
links and Visual Links 142. Such systems may be imple- 
mented as described above, with matching, linked entries in 
Indexed Database 124 and Visual Links Database 140, or 
with text only links stored in Indexed Database 124 and 
Visual Links 142 stored in Visual Links Database 142. In yet 
other embodiments, Server System 114 may be implemented 
with the text portions of all links in Indexed Database 124 
and any corresponding Graphic Icon 144 stored in Visual 
Links 142 with the text and graphics portions linked to one 
another, or Server System 114 may be implemented with 
only a Visual Links Database 140, as described above, but 
with the Graphic Icon 144 component of all text only links 
left empty. 

Finally in this regard, it has been described that in certain 
implementations of Server System 114, Graphic Icons 144 
may be generated by a Visual links Automatic Capture 
Engine 138A residing in a Server System 114 from the text 
and graphics information contained in each HTML 
document, or other type of file that has been accessed by 
Server System 114. Graphic Icons 144 may also be gener- 
ated in a like manner by a user of a Local System 112, using 
an implementation of Visual links Automatic Capture 
Engine 138A residing in the Local System 112, or by the 
operators of a Web Server 118, again using a Visual Links 
Automatic Capture Engine 13 8A. It will also be noted that 
Graphic Icons 144 need not be generated from an HTML 
document by a Visual Links Automatic Capture Engine 
138A, but may be generated independently by a user of a 
Local System 112 or the operators of a Server System 114 
or a Web Server 118, or even by an independent supplier of 
pre -made Graphic Icons 144 using, for example, virtually 
any of a wide range of well known and readily available 
graphics creation programs, and may be used subsequently 
in the construction of Visual Links 142 or, for example, to 
replace an already existing Graphic Icon 144 associated with 
a Visual Link 142. 

In a like manner, Visual Links 142 may also be generated 
or otherwise created by the operators of Web Servers 118, by 
the operators of Server System 114, or by the users of Local 
Systems 112, or otherwise provided. The Visual Links 142 
so created may be embedded in the HTML documents 
residing in Web Servers 118 or in a Server System 114 or 
Local System 112, or may be stored in indexes or databases 
residing in a Web Server 118, a Server System 114 or a Local 
System 112. For example, a Visual Links 142 may be read 
directly from an HTML document or other file by Network 
Data Collector and Indexing Server 122, stored in Indexed 
Database 124 or Visual Links Database 140, as described 
above, and provided to and stored in a Local Server 112 from 
Server System 114. In further example, a Visual Link 142 
may also be read directly by Local Server 112 when the 
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document or other file is accessed by Local Server 112, 
typically by the user "clicking" on a representation of the 
Visual Link 142 embedded and displayed in the HTML page 
or file, as described below, in the manner usual for HTML 
5 links. 

Considering Visual Links 142 further, and assuming the 
use of HTML documents other forms of files supporting 
linking, such as files generated or operated upon application 
programs supporting Microsoft Windows Object Linking 
10 and Embedding (OLE), a Visual Link 142 can be inserted 
into an HTML document by means of an OBJECT Tag or an 
EMBED Tag 146, depending upon whether Netscape of 
ActiveX standards are followed. As illustrated in FIG. 1C, 
the OBJECT or EMBED Tag 146 follows an IMG Tag 148 
35 displaying a Visual Links Symbol 150 and is hyperlinked 
152 to the URL 154, and associated link related information, 
so that a user "clicking" on Visual Links Symbol 150 will 
access the URL 154 and associated information. In an 
alternate implementation, alternate hyperlinked text or 
20 graphics, such as a Visual Link 142 may be included with the 
OBJECT or EMBED tag. 

In this regard, it will be appreciated by those of ordinary 
skill in the relevant arts that there are a number of ways for 
a user to interact with a Visual Links Symbol 150. For 
example, when a user passes a mouse cursor over a Visual 
Links Symbol 150 or icon, the symbol or icon may be 
replaced by the URL or by key words or by a descriptive text 
so long as the cursor is colocated with the symbol or icon, 
or this information may be displayed in a separate window, 
such as a "pop-up" window. 

Referring now to Local System 112, as shown in FIG. 1A, 
Local System 112 is, for example, a personal computer 
executing the Microsoft Windows operating environment 
35 and including typical components, such as a Disk Drive 
156 A and a Memory 15 6B for storing programs and data, a 
Processor 158 for operating on the data under control of the 
programs, and a Display 160 for providing a visual display 
of the operations of the system. Although not shown explic- 
4Q itly in FIG. 1A, it will be understood by those of ordinary 
skill in the relevant arts that Server System 114 and Service 
System 116 will likewise include a Disk Drive 156A and a 
Memory 156B for storing programs and data and a Proces- 
sor 158 for operating on the data under control of the 
45 programs and will generally include a Display 160 for 
providing a visual display of the operations of the systems. 

As also shown in FIG. 1A, Local System 112 further 
includes a Web Access Program 162, such as Netscape 
Navigator, as will Server System 114, although in a different 
50 form and implementation and capabilities from that in Local 
System 112, and, according to the present invention, a Visual 
Links Control 164 and a Visual Links Organizer 166, both 
of which are associated with and operate in association with 
Web Access Program 162. Associated with Visual Links 
55 Control 164 and Visual Links Organizer 166 is a Visual 
Links Library 168 for storing Visual Links 142. Local 
System 112 may also include a Visual Links Automatic 
Capture Engine 138Aor 138B, perhaps in more limited form 
than that implemented in Server System 114, thereby allow- 
60 ing a user of Local System 112 to create Visual Links 142 
locally, rather than only downloading Visual Links 142 from 
Server System 114 or Web Servers 118. 

Visual Links Control 164 operates to interface and inte- 
grate Visual Links 142, Visual Links Organizer 166 and 
65 Visual Links Automatic Capture Engine 138A or 138B, if 
present, into Web Access Program 162, and allows a user to 
access the target of a Visual Link 142 and to save a Visual 
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Link 142 in Visual Links Library 168 or a file. Visual Links representing both Visual Links Automatic Capture Engine 

Organizer 166, in turn, allows a user to create and edit Visual 138A and at least certain aspects of the embodiments of a 

Links 142, to store Visual Links 142 in Visual Links Library Visual Link Manual Capture Engine 138M, is to extract 

168, and to otherwise manage Visual Links 142 and Visual graphics information from the text and graphics presented 

Links Libraries 168. Visual Links Organizer, which will be 5 °n a P&g e of an HTML document, or other file containing 

described in detail further below, also allows a user to g ra P hic information, to be incorporated into a Visual Link 

organize the display of Visual Links 142 in a Visual Links ]* 2 a ? a . Graphic Icon 144 As will be seen in the following 

Library 168 description, and because the icons generated by Visual Link 

. „ , _ „_ , Capture Engine 138 are formed of graphics information 

As shown in FIG. 1A, a Local System 112 may also extracted from the HTML page or document referred to by 

mch.de a Visual Links Screen Saver 170 which is connected » ^ visual ^ U2 me Graphic Icon 144> the 

from Visual Links Library 168 and which is responsive to icQns of me Qt i[lvention tovide visuaUy strong ^ 

the operations of Local System 112 in the usual manner of tmc , ive jm , 0 be used k vjsuaU identifying lhe 

screen savers to generate a screen saver display on Display di HTML page or document to a user. In addition, 

160. The design and operation of screen savers is well and as wiu ^ be seen from ^ following) while no , m 

known and understood by those of ordinary skill in the " itself a ressed imagc> a Graphic Icon 144 may rcquire 

relevant arts and, as such, will not be discussed further significantly less space to store ^ lhan do the 

herein. It must be noted, however, that m accordance with ical ressed " thurnbnail " j^ges 0 f the prior ^ and 

the present invenuon the visual elements of &e screen saver less netWQrk ^ tQ eomaaaiGt!to ^ one system to 

display are comprised of Graphic Icons 144 from Visual anomer . Further, the Graphic Icons 144 of the present image, 

Links Library 168, and may be selected by Visual Links *> bd atcd from th e nchvarfety 0 f images available on 

Screen Saver 170 at random or by user control and I may, for HmL documcnts or agcs? or m other graphics fi i cs> 

example, include the most frequently used Graphic Icons idf . a ^ varif _ of jnu ^ QQt genctally avail . 

144 or selected layouts of Graphics Icons 144, as will be aWe fa tfae sUmd!lTd types of «< icons » used> for example> m 

described in a following discussion. In addition, the usual Microsofl windows ^cations, in part because of the 

operation of a screen saver utility in disabling the screen 25 s]ighay larger size of Graphic Icons 144 and in part because 

saver and returning the display presented on Display 160 to of ^ relative casc ^ which disdnct G Mc 

the executing operations of the system upon, for example, Icons 144 afc gcnerated 

any mouse input by a user may be selectively disabled, ^ of yisual Link c ture 138 

thereby allowing the mouse to be used in conjunction is the ... . e , . . c . • * 

u-t 1JiL Tr it - i o o ™ is the extraction of graphics information from the graphi- 

display of Graphic Icons 144 by Visual Links Screen Saver. 30 . , , 6 . r A . . . %mu 

™f J • * it * i * « i- cally represented text and images present m an HTML 

This operation, in turn, allows the user to select, or ' cIick , j * *l u * ■ ci j* L 

T \aa^ • j-1 jt.^/- it*i<- document or page, or other form of image file, and the 

a Graphic Icon 144 being displayed by Visual Link Screen , . c r . . . . c *• • * ^ u* t 

«n • ^ T ; j , A « j forming of that graphics information into a Graphic Icon 

Saver 170, in the same manner as Web Access 162 and D r 

Visual Link Organizer 166, to invoke Web Access 162 and ' , T . , ^ ^ r 

to access the site, document or file associated with the 35 . Visual Lmk Cap mre Engme 138 may perform this opera- 
selected Graphic Icon 144. Uon m a number of wa >^' for cxam P le > providing a user with 
t . .„ . . , __ _ . . .„ a mechanism to manually delineate and copy an area of an 

Lastly, it will be noted that FIG. 1A illustrates additional { such as an afea of ^ fjTML page or document or 

elements^ the present invention, specifically a Layout othef form of hic { tQ rfse the hic infor . 

Table 412 and a Display Generator 414 in Visual Links m ^ Qn ^ {s ^ tQ ^ & ^ hic Icon U4 Su(± 

Organizer 166, which will be discussed in a later description mechamsms are co mmomy known and implemented using, 

of the generauon and use of display layouts of Graphic Icons fof c ^ ^ h nctions and operations available in the 

144 and other displayable visual objects. Microsoft Windows operating environment and used, for 

Finally, it must be noted that the functions and operations example, in Microsoft Paint and similar drawing programs, 

of Visual Links Control 164 and Visual Links Organizer 166 45 This approach may be used, for example, in an implemen- 

are implemented through the standard functions and utilities tation of Visual Link Capture Engine 138 used in Local 

of Microsoft Windows in the presently preferred embodi- System 112, and designated in FIG. 1A as Visual Links 

ment. The functionality and capabilities of Microsoft Win- Manual Capture Engine 138M, where a user is creating 

dows are well known to those in the relevant arts and are Visual Links 142 for a relatively small number of HTML 

well described in readily available publications, such as 5Q documents or pages or a relatively small number of other 

"Windows 3.1 Programmer's Reference" by James W. image files. 

McCord and published by Que Corporation and "Inside A ^ may ^ usc the facilitics of a graphic paint or 

OLE 2" by Kraig Brockschmidt and published by Microsoft annotation system to add, replace, or delete, in any 

Press, which are incorporated herein by reference as neces- combination, or otherwise modify, selected parts of a 

sary. As such, the following will describe Visual Links 55 selected area A ^ may ais0j ^ aQ alternative or in 

Control 164 and Visual Links Organizer 166 only as nec- combination with such operations, create a part of or the 

essary to describe and understand, for example, the opera- entirety of the visual image by the graphics operations of 

tion of Visual links Automatic Capture Engine 138 A and of such a graphics application program or utility, or by selec- 

Visual Links Organizer 166 in organizing the display of tion of a part of or me entirety of thc visual image from an 

Visual Links 142, which will be described in detail. m independent library of images, and may modify an image or 

_ ^ . ir -i . A . a part of an image selected from a library of images for such 

2^General Description of Visual Uaks Automatic ^ Insuch cases, the image part of a Visual Link 142 

P 1 Str f 1 a 1 nTi Ll ^!^anizer fe thereby ^ ^ h ^ nQ{ ^ from ^ 

166 (FIGS. 1A-1D, 2A and 2B) of aQ KmL image 

As described above, a primary function of Visual Links 65 In this regard, it is discussed in the following detailed 

Automatic Capture Engine 138, hereafter referred to gener- descriptions of a Visual Links Automatic Capture Engine 

ally as a Visual Links Automatic Capture Engine 138 and 138 that it has been found preferable, in the presently 
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preferred embodiment, that the dimensions and aspect ratios Such visual links may, for example and in one possible 
of the Graphic Icons 144 be selected to allow efficient implementation, be held only temporarily, for example, for 
packing into a given display area For these reasons, the the duration of a session, thereby providing a visual refer- 
dimensions and proportions of Graphic Icons 144 are pref- ence for revisiting a server that was accessed earlier in the 

erably selected from a relatively small set of possible 5 session. 

dimensions and aspect ratios to allow the efficient [n a Wq m [{ be fera51e t0 ide an 
organization, packing and dispky of Graphic Icons 144 in a implementation of Visual Links Manual Capture Engine 
directory window. The method for manual selection of an ^J.., „ e . ^ A f , *\ , u p 

area of an image to be used in generating a Graphic Icon 144 * 38M in Server System 114 for use by the operators thereof, 

thereby preferably represents to the user not only the origi- in for example when a source image or page has no suitable 

nal image, or a section of interest thereof, and the area 10 areas from which Visual Links Visual Link Capture Engine 
selected by the user for possible use in generating the 138 can generate a suitable Graphic Icon 144 or when the 
Graphic Icons 144, but also the possible Graphic Icons 144 results generated by Visual Links Visual Link Capture 
that would result from the proposed selection. Engine 138 are ambiguous. Such results may occur, for 

For example, the user may be provided with a mouse example, when a page or image is entirely comprised of low 

controlled "stretchy rectangle" to use in selecting a portion 35 contrast visual structures or when all of the visual structures 
of the original image for possible use in generating a therein are comprised of fine detail text or graphics. 
Graphic Icon 144, in the manner well known, for example, One solution to this problem is, and for example, to use 

in Microsoft Windows graphics programs such as the as the "image" a few words or a legible part of the URL 154, 
Microsoft Paint program. The Graphic Icons 144 generation such as me str j ng 0 f characters typically following W. 

program would then display one or more representations of 20 ^ ^ ^ es 

the Graphic cons 144 that could result from the use of the automatkall it be advantagco J to disp f ay the P docu . 
area delineated by the user, thereby allowing the user to 4 J ' J . , ,. & * i u 

select one of the possible Graphic Icons 144 that would mcnt . or w * b P?«? in a selecUve wa * e * am P 1( ;> b ? 
result from that selection. In another embodiment, and as omittlD S P arts ° f the P a S e comprising small text or material 

illustrated in FIG. ID, the user would use, for example, the 25 otherwise not directly and sufficiently descriptive of the 
mouse cursor to designate one comer, such as the upper left P a S e > as advertisements. Such parts of a page may be 
comer, of an area of the original image and the Graphic Icon °mitted b V use of me methods described in the following for 
144 generation program would overlay and display on the selecting areas of an image to be used in creating Graphic 
original image a set of rectangles representing the set of IcoDS 144 > for example, and advertisements or similar areas 

possible Graphic Icons 144 wherein the set of rectangles 30 of a P a S e ma y be omitted based u P on their non-image 
conform to the range of dimensions and aspect ratios contents, such as links to other sites, 
selected for the Graphic Icons 144, the upper left comer of A Visual Links Visual Link Capture Engine 138 for 
each of the displayed rectangles being located at the initial selecting areas of graphics images and forming them into 
point designated by the user. The user could then select one Graphic Icons 144 must meet a number of requirements and 

of the displayed rectangles as delineating the area of the 35 must provide a number of functions. For example, the Visual 
original image to be used in generated the Graphic Icon 144, Link Capture Engine 138 must provide an evaluation pro- 
for example, by "clicking" on the lower right comer of the cedure and mechanism for comparing the worth, that is, the 
chosen rectangle . visual strength, of various proposed regions of an image, and 

As illustrated in FIG. ID for an exemplary at least affirming the validity of a region of an image as a 

implementation, the upper left comer of the delineating 40 source of information for a Graphic Icon 144. The 

rectangle, designated in FIG. ID as comer A, is first posi- evaluation procedure will appropriately weight and balance 
doned so as to identify the upper left of the intended area. the image areas, bold text areas and logos, and will be 
Thereupon, in the present example, four candidate aspect relatively insensitive and unresponsive to areas containing 
ratio rectangles are displayed superimposed on the image, small fonts and fine details. 

the bottom right comers being designated in FIG. ID as 45 The Visual Link Capture Engine 138 will also provide an 
comers B, C, D and E. The user then selects one of the efficient mechanism for searching an image, such as an 
candidate aspect ration rectangles by "clicking" on the HTML page or document or other form of image, and 
bottom right comer of the selected candidate aspect ratio identifying possible areas or regions to be evaluated or 
rectangle, whereupon the unselected rectangles cease to be selected for use in creating the Graphic Icon 144. In this 

displayed. The user may then re -position the entire selected 50 regard, a given image may contain an extremely large 
candidate aspect ratio rectangle to select the most appropri- number of possible subregions which may serve as source 
ate area of the image by "dragging" the upper left comer of regions for a Graphic Icon 144; for example, a 640 pixel by 
the selected rectangle, and may resize the selected rectangle 1480 pixel bitmap image may contain more than 10 billion 
to fit the most appropriate area by dragging the bottom right possible regions that might be used to generate a Graphic 

comer of the selected rectangle, with the system enforcing 55 Icon 144. 

the aspect ratio constraint for the selected rectangle. In addition, the original image may appear as or be 

visual Link Capture Engine 138 as implemented in Server comprised of a set of definitions of graphic objects, as may 
System 114, however, will preferably provide an automatic be defined, for example, by hypertext markup language 
mechanism, designated in FIG. 1A as Visual Links Visual (HTML). In these instances, it may be desirable to cull the 

Link Capture Engine 138, for selecting areas of graphics 60 graphics objects for the purpose of omitting from the "origi- 
images and forming them into Graphic Icons 144 because nal" image, and thus from consideration for use in a Graphic 
me large number of Visual Links 142 that must be generated. Icon 144, such parts of the "original" image as are corn- 
It may also be desirable to provide an implementation of monly not suitable for use in creating a Graphic Icon 144. 
Visual Links Visual Link Capture Engine 138 for use in Examples of such may include fine text, a wallpaper 

Local System 112, however, for example, when a user 65 background, or bitmap parts that may with high probability 
generally lacks the skill, interest or time to create Visual be automatically classified as advertisements or other pic- . 
Links 142 manually. torial subparts irrelevant to the purpose of visually depicting 
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the essential visual appearance of the original image. Again, 
such parts or components of an original image may be culled 
based upon the methods described herein, or upon their 
contents. 

Finally, the image information extracted from a graphics 
image, which may be of various dimensions and 
proportions, must be shaped into a Graphic Icon 144. In this 
regard, however, a single fixed aspect ratio, for example, of 
3:4, for either the extracted graphics information or the 
Graphic Icon 144 is too constraining considering the pos- 
sible areas or regions that may be selected from a given 
graphics image as most distinctive and most unique to that 
graphics image, both in itself and relative to other graphics 
images. It is preferable, however, to efficiently organize and 
display the maximum number of Graphic Icons 144 in, for 
example, a directory window, and, for this reason, the 
dimensions and aspect ratios of the Graphic Icons 144 will 
preferably be selected to allow efficient packing of any 
arbitrary set of icons into a given display area. For these 
reasons, the dimensions and proportions of Graphic Icons 
144 are preferably selected from a relatively small set of 
possible dimensions and aspect ratios to allow the efficient 
organization, packing and display of Graphic Icons 144 in a 
directory window. 

Visual Link Capture Engine 138 will therefore provide an 
Visual Link Capture Engine 138 that automatically shapes 
the extracted image information into a Graphic Icon 144 
selected from a relatively small set of possible dimensions 
and aspect ratios that are selected for efficient packing into 
a display area 

In a related problem, Visual Links Organizer 166 will 
provide a mechanism for efficient packing any arbitrary 
collection of Graphic Icons 144 having various dimensions 
and aspect .ratios from that set of dimensions and aspect 
ratios into a display area 

First considering the evaluation procedure and mecha- 
nism for comparing the worth, that is, the visual 
distinctiveness, of various proposed regions of an image, 
and affirming the validity of a region of an image as a source 
of image information for a Graphic Icon 144, it is apparent 
that bold text and pictorial material, that is, large or highly 
contrasting features or areas of text or image, are signifi- 
cantly more visible and distinctive than are areas of, for 
example, fine text or low contrast pictorial material. 

According to the present invention, as and described in 
further detail below, the most visually significant features of 
an image, that is, the largest and highest contrast or highest 
visibility features, are emphasized, and the less significant 
features suppressed, by coarsening and blurring the overall 
image, which is usually in the form of a bitmap image. Also 
according to the present invention, the features or regions of 
most significance in an image will generally be as evident in 
a gray-scale version of a bitmap as in a color version, so that 
a gray-scale version of the image may be used rather than the 
color version to simplify and speed the task of identifying 
the regions or features of most significance. 

Pursuant to evaluation, therefore, a coarse grayscale ver- 
sion of the original image, for example, Vi the linear size in 
pixels of the original image, is low-pass filtered, and an 
"edge" image is made from the coarse grayscale version of 
the original image, wherein the edge image is regions having 
high numerical values in the regions of high no n -uniformity 
of the filtered image. The amount of blurry "edge" in any 
subrectangle, relative to its size, is taken as an approxima- 
tion of the degree to which this rectangular area contains 
visually bold or significant information. 
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It is anticipated that some very small regions of the coarse 
grayscale version of the original image may contain high 
evaluations of edge-per-area, yet not be suitable for use as 
Graphic Icons 144. Therefore, and according to a presently 

5 preferred embodiment of the invention, candidate areas are 
required to be of predetermined minimum size and the 
criteria by which regions are rejected for consideration 
includes size. As such, and in the present implementation, if 
two areas exhibit the same average amount of edge-per- 

10 pixel, then the larger of the two is deemed to be the better 
region to be used in creating the Graphic Icon 144. 

Because the Visual Link Capture Engine 138 searches for 
bold and coarse features, and because, for reasons discussed 
in the following, it is not necessary to locate the boundaries 

15 of the Graphic Icon 144* s source area in the original image 
or in the coarse grayscale version of the original image with 
pixel accuracy, the Visual Link Capture Engine 138 per- 
forms a first search with a second, very low-resolution 
image. The very low resolution image is constructed, for 

20 example, made from the V£ size edge image and is, for 
example, Vis the linear size of the original image, or V* the 
size of the Vi scale edge image. The use of such a low 
resolution image, have a proportionally smaller number of 
total pixels and a correspondingly smaller number of pixels 

25 in each sub-region thereof, the Visual Link Capture Engine 
138 can perform a relatively thorough search, involving 
crude test-for-appropriateness of a very wide range of sizes, 
locations and aspect ratios of initially selected regions of the 
original image. Subsequent refinement of the area or areas 

3 q thus found is then based on the more accurate l A size edge 
image. 

Once a suitable area of the Vi size edge image is found, the 
candidate area, which in the present embodiment of the 
invention is a rectangle, is refined by the process of itera- 

35 tively pushing one or another side of the rectangle outward 
by one row of pixels so long as this increases the rectangle's 
worthiness for use in creating the Graphic Icon 144, and 
finally adjusting each side of the rectangle inward or out- 
ward accordingly as the rows of pixels just inside or just 

4 0 outside of the rectangle have or do not have significant edge 
indications. That is, a side of the rectangle is retracted the if 
the strip of pixels inside that side contains little or no edge 
feature and the side of the rectangle is extended if the edge 
indications in the strip of pixels outside that side indicate the 

45 continuation of features already partly contained in the 
rectangle. 

Finally, it has been described above that the rectangle of 
image information extracted from a graphics image as 
representing the visually most distinctive and unique area of 

50 the original image may be of various dimensions and 
proportions. It has also been described that the selected 
rectangle of graphic information is then preferably shaped 
into a Graphic Icon 144 having a size and aspect ratio 
selected from a relatively small set of possible dimensions 

55 and aspect ratios to allow the efficient organization, packing 
and display of Graphic Icons 144 in a directory window. 

In a presently preferred embodiment of the invention, the 
set of possible aspect ratios for the final Graphic Icons 144 
have been chosen to include aspect ratios of 6x2 (units of 

60 height to width), 4x3, 3x4 and 2x6 as these aspect ratios 
result in Graphic Icons 144 having the same total area and 
because an arbitrary group of Graphic Icons 144 having 
these aspect ratios will pack efficiently into a given display 
area. The ultimate icons will therefore be resized, according 

65 to one exemplary set of sizes, to be 120 pixels high by 40 
pixels wide, or 80 pixels high by 60 pixels wide, or 60 pixels 
high by 80 pixels wide, or 40 pixels high by 120 pixels wide, 
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or they may be slightly less in one or both dimensions but images, that is, the relative "brightness" of image 

will be deemed nominally to "occupy" these screen display components, in identifying those areas of an original image 

areas. Accordingly, and because the region of the original most suitable for incorporation into a Graphic Icon 144. As 

image that has been selected as visually most distinctive and such, the following descriptions of Visual Link Capture 

unique will rarely have one of these predetermined aspect 5 Engine 138 will primarily focus on the grayscale implemen- 

ratios, a final adjustment of the area or region of graphic tation of the processes for generating Graphic Icons 144. 
information that has been selected and extracted from the Alternate implementations of the Visual Link Capture 

original image is necessary to conform the selected area or Engine 138, however, may utilize the color information, that 

region to one of these predetermined aspect ratios. is, such as the red, green and blue components, of Original 

A number of such adjustment processes are possible, and 30 Image 210 in identifying the visually most significant and 

the presently preferred embodiment of the Visual Link unique region of the image. The methods, processes and 

Capture Engine 138 may or does use any or any combination mechanisms of the capture engine of the present invention as 

of these adjustment processes. For example, in a first implemented for the use of color information are very 

process, the icon may be left in the shape as extracted from similar to those for the use of grayscale images, however. As 

the original image and placed in a final display into the 15 such, the use of color image information will be discussed 

best-fitting shape of area, leaving blank unused area on left briefly in the following, using FIG. 2A for purposes of 

and right, or on top and bottom. In a second process, the icon illustration, before continuing with a detailed discussion of 

can be distorted by a vertical or horizontal stretch so that it the mechanisms using grayscale information and those of 

fits such an area and, in a third process, the area from which ordinary skill in the relevant arts will understand the adap- 

the icon is cut may be extended either vertically or horizon- 20 tation of the grayscale mechanisms to the use of color 

tally to match an acceptable shape while, in a fourth process, imf ormation. 

a combination of stretching, of up to, perhaps, 10%, may be In a presently preferred embodiment of a Visual Link 

combined with an extension of the source area. Finally, in Capture Engine 138 using color information, the colors are 

yet another method, and if and where a background color not treated separately through the entire process. Instead the 

can automatically be discerned, that color may be used to fill 2 $ color information is included or combined into a single edge 

the extra area so that the icon's sides do not make a image corresponding to and equivalent to Va scale Filtered 

disconcerting cut through other visual objects in the vicinity. Grayscale Image 214 and this color Filtered Grayscale 

In all cases, therefore, the Graphic Icons 144 are produced Image 214 is then used in the processes and mechanisms 

by appropriate scaling, so that they each occupy the same described in the following in the same manner as a grayscale 

amount of screen area and pack efficiently into the display 30 Filtered Grayscale Image 214. 

screen area, such as directory window. In addition, and as As will be described in the following with regard to the 

discussed in the following, the Graphic Icons 144 may be grayscale implementation of Visual Link Capture Engine 

but are not necessarily all created to the same color scheme, 138, the amount of "edge" at a given pixel of an image is an 

such as the 256 color standard used in Microsoft Windows, indication of how much the "brightness" of the image is 

so that they can be displayed simultaneously, and in certain 35 changing at that pixel and is thereby an indication of the 

implementations the final color compatibility may be distinctiveness of that region of the image relative to other 

achieved by a color conversion just preceding the display of regions of the image. In the implementation of Visual Link 

the icons to a user. Finally, and in addition to the above Capture Engine 138 using color information, however, the 

described advantages, each display of Graphic Icons 144, amount of "edge" at a pixel of the image is an indication of 

such as a directory or folder in the directory, will, as a whole, 40 how much the red or green or blue components of the image, 

present a certain distinctiveness because of its near-unique or any combination of the red, green or blue components of 

combination and placement pattern of Graphic Icons 144, the image, are changing in that region, as well as an 

thereby providing the user with a further visual aid for indication of how much the "brightness" of the image is 

looking for, remembering and recognizing individual changing at that region. The amount of "edge" therefore 

Graphic Icons 144. 45 reflects the amount of "color-and -brightness" change, which 

Finally, the various images and versions of images used is generally referred to in brief as the "color" change, 
by the Visual Link Capture Engine 138 of the present This embodiment of Visual Link Capture Engine 138 

invention and described above are illustrated in FIG. 2A thereby determines a "color change" at each pixel of the 

wherein Image 210 represents the original image of an blurred image wherein the color change may be usefully 

HTML page or document, or other graphics image, and 50 defined and used as any of a number of measures. For 

Grayscale Image 212 represents the Va size grayscale image example, the color change may be defined and determined as 

generated from Original Image 210 and Filtered Grayscale the red component change plus the green component change 

Image 214 represents low pass filtered, that is, coarsened plus the blue component change plus the brightness com- 

and blurred image generated from Grayscale Image 212 and ponent change at a pixel, wherein the brightness component 

Edge Image 216, in turn, represents the edge image gener- 55 change is the only component considered in the presently 

ated from Filtered Grayscale Image 214. Coarse Edge Image preferred grayscale implementation of the capture engine 

218 then represents the Vi6 scaled image generated from described below. In one implementation, the red component 

Edge Image 216 for purposes of identifying candidate change at a pixel may, for example, be determined as the red 

regions from which to construct the Graphic Icon 144, while component left-right change across the pixel plus the red 

Graphic 44 Image 220 represents the region of Original 60 component above-below change across the pixel, and lik- 

Image 210 that has been selected, as described above, for wise for the green, blue and brightness components. The 

construction of the Graphic Icon 144, and Graphic 44 left-right and above-below changes at a given pixel and for 

represents Graphic 44 Image 220 after adjustment to one of a given component may, in turn and for example, be defined 

the predetermined sizes and aspect ratios chosen for Graphic as the absolute values of the differences between that com- 

Icons 144, that is, Graphic 44 is the final Graphic Icon 144. 6 5 ponent at the pixels to, respectively, the left and right and 

It will be noted that the presently preferred embodiment above and below that pixel. In yet other implementations, 

of Visual Link Capture Engine 138 operates with grayscale the change in a component may be defined as either the 
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larger of the absolute value of the difference in the compo- The capture mechanism would then perform the above 

nent between the top and bottom neighbors of each pixel or analysis of each candidate region, but three times, once for 

the absolute value of the difference in the component each color component of each candidate region, and com- 

between the right and left neighbors of each pixel, thus bines the three evaluations for each candidate region to 

separately considering changes occurring in the horizontal 5 generate a value representing the visual merits of each 

and vertical directions. candidate region. The Visual Link Capture Engine 138 then 

The "amount of edge" at given pixel, in turn, may be selects the visually most significant of the candidate regions 

determined as the average, for that pixel, of each of the to use in constructing the Graphic Icon 144 as described 

component changes and, in another implementation, as the above. 

maximum of the component changes, rather than the aver- 10 Returning to consideration of the operation of Visual Link 

age. Capture Engine 138 using only brightness information, the 

Whichever method is used, this process thereby deter- above described capture of image information and creation 

mines for, each pixel, a value depending upon both how of a Graphic Icon 144 can be illustrated by way of an 

much the brightness component is changing in a given example. In this instance, it is assumed that the Original 

direction at each pixel and how much the red, green and blue 15 Ima S e 21° exists m one of the more complex image formats, 

components are changing in that direction at each pixel. For sucn as tnc 256-color bitmap image format wherein each 

example, given a "pie chart" wherein the different sections bvte value °~ 255 corresponds to a color whose red, green 

of the chart have approximately the same brightness but are and °l ue components are defined in an auxiliary color 

drawn in different colors, this process will determine sig- lookup table (LUT). 

nificant "edge" along the radial division lines between the 2 o ^ ne ^ sua l Li Q k Capture Engine 138 first reads the color 
sections of the chart. LUT and composes a gray-scale LUT whereby the bright- 
It will be noted that in yet another alternate implementa- ness of each of the 256 is taken t0 be the a verage of 
tion of the capture engine using color information, Visual that color's red, green, and blue component values. It will be 
Link Capture Engine 138 may generate three Grayscale noted m tms regard that alternative and more refined mea- 
Images 212 wherein each Grayscale Image 212, respectively 2 5 sures of brightness are known to those of skill in the relevant 
designated in FIG. 2A as Red Color Image 212R, Green arts > bul mis method is sufficient for the present embodiment 
Color Image 212G, and Blue Color [mage 212B, contains of me Visual Link Capture Engine 138. Then, and assuming 
the grayscale equivalent values of, respectively, the red, for purposes of the present example that the bitmap corn- 
green and blue components of Original Image 210. The prising Original Image 210 is free of border, marginalia or 
Visual Link Capture Engine 138 will then execute the same 30 omer extraneous accouterments, the Original Image 210 
operations as described above, but for each of the three bitmap is read and processed into a l A size grayscale image 
Color Images 212R, 212G and 212B separately, and will comprising Grayscale Image 212. As such, 116 source pixels 
generate three corresponding Coarse Edge Images 218, one of Original Image 210 contribute to each destination pixel of 
corresponding to each of Red Color Image 212R, Green Grayscale Image 212 and this conversion is accomplished 
Color Image 212G, and Blue Color Image 212B. In one 35 bv adding and averaging the brightness of the 116 source 
implementation, the three Coarse Edge Images 218, which V^els corresponding to each Grayscale Image 212 pixel, as 
could possibly correspond to three different candidate taken from tbe gray-scale LUT. 

regions of Original Image 210, are then evaluated as The low-pass filtered image comprising Filtered Gray- 
described above to select the visually most significant and scale Image 214 is then created from Grayscale Image 212 
unique region of the original image to use in constructing the 40 bv use of a kernel array of coefficients, that is, by multiply- 
Graphic Icon 144 as described above. ing neighboring pixels of Grayscale Image 212 by the 
In one alternate implementation of the three color image numbers of the kernel array and dividing by 18, wherein an 
process, the Visual Link Capture Engine 138 uses the three exemplary kernel array may be represented by 
original Coarse Edge Images 218 corresponding to Red 
Color Image 212R, Green Color Image 212G, and Blue 45 

Color Image 212B to select the region or regions of the 00T00 

original image providing the visually most significant and 01210 

unique region or regions of the original image, each being 12221 

selected on the basis of a single color component, that is, red, 01210 

green or blue. The result may again be one, two or three 50 0 0 1 Q o, 

different candidate regions of the original image, depending, 

for example, on whether any of the single color component and wherein the value 18 is the sum of the multipliers in the 

Coarse Edge Images 218 overlap. The Visual Link Capture kernel. 

Engine 138 then generates a Red Color Image 212R, Green In then creating Edge Image 216, it must be noted that the 

Color Image 212G, and Blue Color Image 212B for each 55 value of each pixel of Edge Image 216 depends on the values 

candidate region selected on the basis of the individual color of the left, right, top and bottom neighbors of the corre- 

components. In most instances, however, the automatic spending source pixel in Filtered Grayscale Image 214. The 

color mechanism need only generate two new grayscale value of each pixel of Edge Image 216 may therefore be 

images for each region as each region will already be found by finding the brightness difference between left and 

represented by one grayscale image. For example, if a given 60 right neighbors and the brightness difference between top 

region has been selected of containing the visually most and bottom neighbors, and then adding the absolute values 

significant area of Original Image 210 based upon the red of these values, multiplying each result by a first constant, 

color component, a Red Color Image 212R and the corre- subtracting a second constant, and clipping the result for 

sponding Filtered Grayscale Image 214, Edge Image 216 each pixel of Edge Image 216 to a minimum value of 0 and 

and Coarse Edge Image 218 will already exist, so that the 65 maximum value of 255. Alternatively, the value of each 

Visual Link Capture Engine 138 need generate this images pixel of Edge Image 216 may be determined as the absolute 

only for the blue and green components. difference between the corresponding left and right vs. top 
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and bottom pixels of Filtered Grayscale Image 214, which 
tends to emphasize horizontal and vertical edges in the 
images. In yet another alternate implementation, the same 
method may be executed, but using the red, green and blue 
color components, as described above, thereby causing the 5 
Visual Link Capture Engine 138 to evaluate the candidate 
regions of the original image on the basis of image structures 
exhibited by differences in color as well as brightness. 

The value assigned to each block, or coarse pixel, of the 
resulting extremely coarse Vie size edge image, that is, JQ 
Coarse Edge Image 218, is then the sum of a corresponding 
4x4 pixel block of Edge Image 216 pixels, these values 
being stored as word -size values rather than byte-size val- 
ues. 

The Visual Link Capture Engine 138 then determines the 15 
rectangular region to be used as source for the Graphic Icon 
144 by executing four distinct steps, or operations: 

(1) an initial rectangular region is found by exhaustive 
search in the very coarse image, Coarse Edge Image 
218; 20 

(2) by reference to the Yi size image, Edge Image 216, this 
area is extended on any or all sides so long as its figure 
of merit increases; 

(3) by iterative successive treatment, each side of the 
grown rectangle is adjusted either in or out as best fits 25 
the edge structure found in the local area (cutting back 

if the rectangle has extended into empty space and 
pushing out if structure seems to have been truncated); 
and, 

(4) the otherwise "final" area is adjusted to conform to one 30 
of the selected aspect ratios to form the Graphic Icon 
144. 

Considering each of these operations further, the mecha- 
nism for examining candidate rectangles will be discussed in 
a following detailed description and it will shown therein 35 
that the mechanism used in the presently preferred embodi- 
ment of Visual Link Capture Engine 138 permits fast sum- 
ming values in candidate rectangles, thereby allowing Visual 
Link Capture Engine 138 to test a large number of candidate 
rectangles quickly and efficiently. As such, the initial can- 40 
didate rectangle is found by exhaustive examination of all 
candidate rectangles in a Coarse Edge Image 218 having 
widths in the range of 6 through 10 pixels less-than- full- 
width pixels and heights in the range of 6 through 10 
less- than -full-height pixels. That is, the starting rectangle 45 
must be at least 6 pixels of the Vie size image and at most 
ten less than the full size rectangle in both the horizontal and 
vertical directions so that the search in the coarse space is 
exhaustive exepct for very thin and very tall or very wide 
rectangles. The evaluation of these candidate rectangles on 50 
the same basis that is subsequently used to evaluate growth 
of the selected rectangle, which will thereby be equally 
exhaustive and with the same possible exceptions. It will 
also be noted that 6-through-10 pixel high by 6-through-10 
pixel wide rectangles of Coarse Edge Image 218 were 55 
selected as providing areas having an acceptable probability 
of containing at least parts of visually significant image 
structures while being sufficiently small as to reduce the 
processing burden. 

As described above, the search for and subsequent modi- 60 
fication of suitable candidate rectangles for the generation of 
Graphic Icons 144 requires a method for the evaluation of 
each rectangle or modification thereof, that is, the determi- 
nation of a "figure of merit" or "figure of assurance" for each 
candidate rectangle. The search for a suitable area of an 65 
image to comprise of Graphic Icon 144, or a proposed 
modification of a candidate rectangle to determine whether 
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the candidate rectangle is improved by moving the sides of 
the rectangle inwards or outwards, results in each case in a 
local maximum in the total search space wherein the local 
maximum has a "value" and a "sharpness" which together 
can be used to express a figure or merit or assurance that the 
rectangle is the or a proper area to be used in generating a 
Graphic Icon 144. In addition to being used by Visual Link 
Capture Engine 138 to evaluate a candidate rectangle or a 
modification to a candidate rectangle, the figure of merit or 
assurance can then be used to flag or otherwise indicate 
results having a low figure of assurance for subsequent 
human approval or correction, for example, by use of a 
Visual Links Manual Capture Engine 138M. 

It will be appreciate by those of ordinary skill in the arts 
that the value component of the figure of merit or assurance 
of a candidate rectangle is a function of the location of the 
candidate rectangle in the image and the size of the candi- 
date rectangle. That is, the location of a candidate rectangle 
determines which area of the image is included in the 
rectangle and, in turn, is a function of the average amount of 
image "edge" in the candidate rectangle, wherein "edge" 
may be defined as the local degree of visual coarse structure 
non-uniformity of image in the selected area of the image. 
The degree of local visual coarse structure non-uniformity, 
in turn, may be generally defined as the amount of contrast, 
or difference, in brightness or color between adjacent areas 
in the rectangle and the "sharpness", or abruptness, of the 
boundary between adjacent areas of different brightness or 
color in the rectangle, that is, how fast does the change in 
brightness or color occur when moving from one area to 
another. 

The criteria of "sharpness" also applies in the steps 
wherein the sides of a candidate are adjusted inwards or 
outwards to incorporate the entirety of a visually strong local 
structure already at least partially included in the candidate 
rectangle into the final candidate rectangle. In this sense, 
"sharpness" is a measure of how fast and in what direction 
the figure of merit or assurance value of the candidate 
rectangle changes per unit distance as a side of the rectangle 
is moved inwards or outwards. 

In this regard, it will be appreciated by those of ordinary 
skill in the arts with respect to the determination of a figure 
of merit or assurance of a candidate rectangle that the step 
of generating Filtered Grayscale Image 214 is essentially a 
pre-processing step with respect to the search for and 
evaluation of candidate rectangle wherein small text and low 
resolution or low contrast areas of the original image are 
discarded as being of little value in generating a Graphic 
Icon 144 presenting strong visual components to a viewer. 

Finally, the size of the rectangle determines the amount of 
original image incorporated into the candidate rectangle and 
in the presently preferred embodiment of Visual Link Cap- 
ture Engine 138 is selected on the general principle that a 
larger candidate rectangle is preferable over a smaller can- 
didate rectangle. 

In the presently preferred embodiment of Visual Link 
Capture Engine 138, therefore, the figure of merit or assur- 
ance is determined as a function of the average amount of 
"edge" per pixel in the rectangle and adding a contribution 
value that increases, up to a limit, with increasing size of the 
rectangle. In the presently preferred embodiment, the actual 
size contribution has two breakpoints with increasing value 
of the perimeter of the rectangle, such that for rectangles 
beyond a certain size there is no further value added for a 
further increase in size of the rectangle. This reduces, but 
does not eliminate, the possibility of very small dispropor- 
tionate structured areas from becoming Graphic Icons 144, 
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and prevents a potential "run- away" condition whereby the 
"figure of merit" increases with the size of the rectangle 
regardless of content. 

The near-final operation of pushing a side of a rectangle 
in or out is performed by computing the amount of edge in 5 
the strip of pixels just inside the rectangle and, if signifi- 
cantly lower than average, excluding the strip from the 
rectangle, that is, pulling the side of the rectangle back, or, 
if the strip just outside the side of the rectangle contains a 
significant amount of edge structure, indicating that image 10 
structure has been truncated that probably should be 
included in the region, including the strip in the rectangle, 
that is, pushing out that side of the rectangle. 

Finally, the final sizing and shaping of the Graphic Icon 
144 area has been discussed above. However, for the case in 15 
which no "filling" or "stretching" is to be done, but the 
source area is made to conform to an acceptable ratio of, say 
6:2 or 4:3 of 3:4 or 2:6, the following are the steps for 
deciding which sides are to be pushed out, relative to the 
existing rectangle. 20 

These "rules" for the steps for final sizing and shaping 
may be represented schematically, including the additional 
in-between ratios of 5:9 and 1:1 and 9:5 as division points, 
by: 

->6:2<-9:5->4:3<-l:l->3:4<-5:9-»2:6<- 25 
The meaning of the schematic is: 
if the height; width ratio is greater than 6:2, widen the area 
to 6:2 

else if the ratio is greater than 9:5, stretch its height to 6:2; 
else if it is greater than 4:3, widen it to make it 4:3; 
else if it is greater than 1:1, make it taller so as to make 
it 4:3; 

else if it is greater than 3:4, widen it to make it 3:4; 

else if it is greater than 5:9, make it taller to make it 3:4; 35 

else if it is greater than 2:6, widen it to 2:6; 

else make it taller to make it 2:6 

It should be noted that in the example described above, 
the final adjustment for a selected aspect ratio has, for the 
sake of illustration, been slightly simplified. For example, in 40 
the most common implementation the final Graphic Icon 
144 will have a thin border area on all sides for the purpose 
of delineating it from neighboring icons in screen layouts 
containing many icons, as illustrated in FIG. 2B. The 
candidate areas are therefore slightly smaller, and ratios 45 
slightly different, if no distortion whatever is to be allowed 
in re-sizing from the source image to the icon, although a 
slight re-sizing may not only be acceptable but even desir- 
able in order to use a more desirably shaped or sized region 
from the source image. 50 

It must also be noted that the "tiling" of many icons of the 
various acceptable sizes and shapes, as illustrated in FIG. 
2B, is a "bin-packing" problem of tiling geometry and, in 
certain implementation, such as with a relatively small 
number of icons per display, for example, 10 to 20, Visual 55 
Links Organizer 166, which will be discussed further in the 
following descriptions, may be implemented by computing 
and tabulating one arrangement for each maximally achiev- 
able icon population distribution. Thus, the possible arrange- 
ments for a display or a sub-section of a display can be 60 
determined in advance, as pre-arranged packing patterns for 
packing, for example, two 6:2s, four 4:3s, three 3:4s and one 
2:6, and stored for use in a display or a sub -section of a 
display as necessary. 

Finally, it must be noted that the Original Images 210 65 
from which Graphic Icons 144 are generated will generally 
be acquired, or captured, by Local System 112 or Server 



,838 Bl 

26 

System 114 as a graphics file or component of a standard 
type, such as a GIF or JPEG image or an HTML document 
or page containing graphic information in the form of GIF 
or JPEG images, and that the graphic file or component will 
most usually be a color image in any of several different 
color information formats. There are many well known and 
available methods and utilities for converting between GIF 
and JPEG images and, while it is preferable that Visual Link 
Capture Engine 138, Visual Links Control 164 and Visual 
Links Organizer 166 operate with a standard image format, 
the conversion between one formal and another is well 
understood by those of ordinary skill in the relevant arts and 
will not be discussed further herein. 

As regards color information formats, it is again prefer- 
able that Visual Link Capture Engine 138, Visual Links 
Control 164 and Visual Links Organizer 166 use a standard 
color format, of which there are several. For example, the 
256 color format as used in Microsoft Windows, provides 
256 numeric values that may be assigned to corresponding 
color values wherein the color of each pixel is coded in one 
byte of information as a RGB (red, green, blue) value and 
wherein each of the 256 different numeric values may be 
assigned to a different combination of red, green and blue 
and thus to a different color. In the "high color" format, each 
pixel's red, green and blue values are stored in 5 or 6 bit 
fields of a 16 bit word, and in the "true color" format each 
pixel's red, green and blue values are expressed directly in 
three corresponding bytes, giving 256 different possible 
values each of red, green and blue. 

It is also preferable that the color format chosen for use in 
Visual Link Capture Engine 138, Visual Links Control 164 
and Visual Links Organizer 166 be compatible with, for 
example, Microsoft Windows and Microsoft Windows com- 
patible application programs, such as Netscape and 
Microsoft Explorer, as it is anticipated that Visual Links 142 
will be used and displayed in such an operating environment 
and with such application programs. For this reason, the 
presently preferred implementations of Visual Link Capture 
Engine 138, Visual Links Control 164 and Visual Links 
Organizer 166 generate, store and display Graphic Icons 144 
in a 256 color format that is compatible with Microsoft 
Windows and Microsoft Windows compatible application 
programs, such as Netscape and Microsoft Explorer. 

In this regard, it should be noted that the 256 color format 
allows each of the 256 different numeric values that can be 
expressed in a single byte to be assigned arbitrarily to a 
different color and that, as a consequence, the 256 colors 
used in Visual Link Capture Engine 138, Visual Links 
Control 164 and Visual Links Organizer 166 need not be 
identical to those assigned to corresponding numeric values 
in Microsoft Windows and Microsoft Windows compatible 
application programs, such as Netscape and Microsoft 
Explorer, so long as the color assignments are sufficiently 
similar to be visually compatible. 

It is possible, however, that a given Original Image 210 
will use a 256 color format in which the color/numeric value 
assignments are visually incompatible with the assignments 
used in Visual Link Capture Engine 138, Visual Links 
Control 164 and Visual Links Organizer 166 or in Microsoft 
Windows and, in these cases, the reassignment of color/ 
numeric values is easily accomplished through a lookup 
table, as well known in the relevant arts. In a like manner, 
an Original Image 210 in the high color or true color formats 
may be readily converted into the chosen 256 color format 
by determining the bit and byte assignments to red, green 
and blue value, unpacking the color information from the 
format into red, green and blue values, and converting the 
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red, green and blue numeric values into comparable red, 
green and blue values in the chosen 256 color format, either 
through lookup tables or, in some instances, by numeric 
calculation. 

Finally in this regard, it should be noted that Visual Link 5 
Capture Engine 138 will perform a conversion of the Origi- 
nal Image 210 from the received color format to the chosen 
256 color format before the Grayscale Image 212 is gener- 
ated from the Original Image 210, so that the generation of 
Grayscale Images 212, and all subsequent operations, 10 
including the final generation of the Graphic Icon 144, are 
performed in a standard manner. As illustrated in FIG. 2A, 
it should also be noted that it may be desirable to display an 
image of the Graphic Icon 144 or potential Graphic Icons 
144 during operation of Visual Link Capture Engine 138, 15 
such as in the implementation of Visual Link Capture Engine 
138 used in Local Server 112 where the user is actively 
interacting with the system. In these applications the pres- 
ently preferred implementation of Visual Link Capture 
Engine 138 generates and displays a temporary representa- 20 
tion of the Graphic Icon 144 or potential Graphic Icons 144 
being considered or generated to the user using the original 
color format of the Original Image 210, thereby avoiding the 
need to perform a color format conversion for a temporary 
display. This image or representation of the Graphic Icon 25 
144 is not stored or saved, however, and only the 256 color 
format final Graphic Icon 144 is stored for later use. 

Lastly, it has been described above that certain imple- 
mentations of Visual Links Capture Engine 138, such as 
Visual Links Manual Capture Engine 138 M in Local System 30 
112, provide a user with the capability to manually designate 
a region of Original Image 210 and to convert the designated 
region of Original Image 210 into a Graphic Icon 144. These 
implementations of Visual Link Capture Engine 138 pref- 
erably utilize the graphic finctions and capabilities provided 35 
in the operating environment in which Visual Link Capture 
Engine 138, Visual Links Control 164 and Visual Links 
Organizer 166 are executing, such as Microsoft Windows. In 
these implementations, the Visual Links Manual Capture 
Engine 138M and Visual Links Control 164 provide inter- 40 
faces to, for example, the Microsoft Windows graphics 
functions which allow the user to delineate and copy a 
selectable rectangular region of an image, as well known and 
understood by those of ordinary skill in the relevant arts. 

The selected region is then provided directly to the Visual 45 
Link Capture Engine 138 final shaping and sizing functions, 
described above and described further below, which perform 
the final steps of generating a Graphic Icon 144 from the 
selected region. It should be noted that, in certain 
implementations, it may be desirable to limit the user's 50 
selection as regards the size and proportions, or aspect 
ratios, of the selected region, as described herein above. 

3. Detailed Description of Visual Link Capture 

Engine 138 55 

Having described the overall operation of a Visual Link 
Capture Engine 138, the following will describe an imple- 
mentation of Visual Link Capture Engine 138 in greater 
detail. It will be noted, in this regard, that certain aspects of 
the structure and operation of Visual Link Capture Engine 60 
138, such as the methods and mechanisms for converting 
between color formats, generating Grayscale Images 212 
from Original Images 210, Filtered Grayscale Images 214 
from Grayscale Images 212, Edge Images 216 from Filtered 
Grayscale Images 214, and Coarse Edge Images 218 from 65 
Edge Images 216, will be understood by those of ordinary 
skill in the relevant arts from the above discussions of an 
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Visual Link Capture Engine 138 and will accordingly not be 
described in further detail except as would assist in under- 
standing the present invention. 

a. General Description of Visual link Capture Engine 138 
(FIGS. 3A-3H) 

Referring now to FIG. 3A1 and 3A-2, the Visual Link 
Capture Engine 138 is implemented in a presently preferred 
embodiment in Local System 112 and in Server System 114 
as a plurality of program modules residing in Disk Drive 156 
or Memory 158 and controlling the operations of Processor 
160 to execute the operations described herein on image data 
residing in Disk Drive 156 or Memory 158. As is well 
known and understood in the relevant arts, such programs 
control the operations general purpose Processor 160 and its 
associated components to operate as one or more dedicated, 
single purpose mechanisms for performing specific opera- 
tions that are determined by the programs. Finally, it is 
assumed for purposes of the present discussions that Server 
System 210 and Local System 212 are operating under the 
control of and incorporate the functions and capabilities of 
a well known and understood operating system such as 
Microsoft Windows or the general equivalent providing 
object based processing, object linking, and a variety of 
functions and operations that are generally available to the 
program creator, such as graphics editing functions. 

The organization of Visual Link Capture Engine 138 as a 
group of processor control program modules and the func- 
tional mechanisms provided by each of the program control 
modules is illustrated in FIG. 3A and in the exemplary 
program listings provided in Appendix A. 

Before beginning the following detailed description, 
however, certain aspects of the control program listings 
provided in Appendix A should be understood, as should 
certain technical details and assumptions of the following 
descriptions. First, the control program listings provided 
include certain versions of the program modules having the 
extension ".noc". These listings contain alternate implemen- 
tations of certain program modules that may be used in 
alternate embodiments of the present invention, and imple- 
mentations that, for example, provide displays that are not 
necessary for essential operation of the present invention but 
that provide displays that may be advantageous for a user or 
for one who maintains the programs. 

For example, there may be implementations in which 
immediate visual feedback to a user is desired, wherein the 
Visual Link Capture Engine 138 operates first to select a 
potential Graphic Icon 144 region of Original Image 210 and 
presents the selection as a suggestion that the user may often 
accept without revision. In other instances the user may 
choose to intervene and, taking some time and care, adjust 
the source region from Original Image 210 or pick an 
entirely different source region from Original Image 210. 
Other alternate embodiments illustrated in these listings 
utilize red, green blue color information in selection of 
regions of Original Image 210 to convert to a Graphic Icon 
144, again as discussed above. 

In addition, the following descriptions assume that Origi- 
nal Images 210 are in the form of ".bmp format" screen 
captures of a 640-pixel- wide by 480-line-high screen, used 
in color-mapped mode with 256 colors. In the following 
examples, only 408 of the 480 lines are actually used, and 
the colormap is assumed to start in byte [54] of the file, each 
entry consisting of the four bytes: blue value, green value, 
red value, zero. It is also assumed that, immediately there- 
after (starting in byte [1078]), is the byte-per-pixel data, 
appearing bottom line first. It is further assumed for pur- 
poses of the following discussions that such files, or RAM 
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bitmap equivalents, will be available without marginal 
frames, Windows buttons, sliders, etc. In the exemplary 
embodiment presented herein, such marginalia are stripped 
by strictly ad hoc means wherein for each image there is side 
information describing how the picture is to be stripped — by 5 
replacing a certain amount of the side material with addi- 
tional white or black background. It will be understood by 
those or ordinary skill in the relevant arts that these and other 
techniques may be used to obtain Original Images 210, and 
how the system described herein may be readily adapted to 10 
other image formats and to other technical assumptions. 

As illustrated in FIG. 3 A, the control programs compris- 
ing Visual Capture Engine 138 include a Main Program 310 
which feeds information about images, one at a time, to an 
image processing mechanism comprised of a Processing 15 
Program 312 and a Find Area Program 314. As will be 
described in further detail below, Processing Program 312 
generates Grayscale Images 212 from Original Images 210 
and passes Grayscale Images 212 to Find Area Program 314, 
which generates Filtered Grayscale Images 214, Edge 20 
Images 216 and Coarse Edge Images 218 and identifies 
appropriate areas of Original Image 210 to iconize, that is, 
to convert into Graphic loons 144, by searching for locations 
of bold image, that is, areas of Original Image 210 holding 
large fonts, photos, logos or combinations of these. 25 
b. Processing Program 312 — Grayscale Image Generator 
316 

For the reasons stated above, and for reasons of speed, the 
search performed by Find Area Program 314 is based on one 
or more small versions of Original Image 210 and, in the 30 
present implementation, the search is performed using a Y* 
size gray-scale images, that is, Grayscale Images 212, gen- 
erated by a Grayscale Image Generator 316 implemented by 
Processing Program 312, and derivatives of Grayscale 
Image 212, that is, Filtered Grayscale Image 214, Edge 35 
Image 216 and Coarse Edge Image 218, generated by Find 
Area Program 314. 

In alternate embodiments, however, the search process 
may be performed with images having less or more 
reductions, depending upon the processing power and speed 40 
of Local System 112 or Server System 214 and the time 
available to process each image. It should be noted, in this 
regard, that the loss of quality of result appears not be 
significant for greater reductions in image size as the object 
of the search is only to identify regions of strong, bold 45 
image. Larger images are desirable, however, at least in 
those implementations providing the user with a display of 
the proposed Graphic Icon 144 areas under consideration, as 
providing a more legible and detailed image to the user, and 
the modifications to Processing Program 312, and to Find 50 
Area Program 314, to generate and use images having 
greater or lesser reductions will be well understood by those 
of ordinary skill in the relevant arts. 

It should also be noted, as discussed above, that Process- 
ing Program 312 and Find Area Program 314 may be 55 
implemented to use the red, green and blue color compo- 
nents of Original Image 210, as well as the brightness 
component, in searching for and identifying the areas of bold 
image suitable for Graphic Icons 144 and, in this 
implementation, Processing Program 312 may generate, for 60 
example, Y* scale grayscale images of the red, green and blue 
components of Original Image 210 for use by Find Area 
Program 314, and for display to the user. 

It has been found, however, that acceptable results are 
normally obtained from brightness information alone. As a 65 
consequence, in the presently preferred embodiment, Pro- 
cessing Program 312 transforms the RGB (red, green, blue) 
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Original Image 210 into a grayscale image, as discussed 
previously, by using a Color Lookup Table 318Ato compose 
a Greyscale Lookup Table 318B which is used, for each 
pixel in an Original Image 210, to determine greyscale 
equivalent values for the red, green and blue values associ- 
ated with each of the 256 numbers. A Grayscale Converter 
320, in turn, receives these values and sums these values for 
4x4 blocks of Original Image 210 pixels to generate a 
"Brightness" Value 322 for each 4x4 block of Original 
Image 210 pixels, the resulting Brightness Value 322 for 
each block becoming the grayscale value for a correspond- 
ing single pixel of the corresponding Vi scale Grayscale 
Image 312. It will be appreciated by those of ordinary skill 
in the arts that Grayscale Converter 320 may generate 
Grayscale Images 212 at sizes other than Y* scale by corre- 
sponding changes in the numbers and arrangements of the 
Original Image 210 pixels selected to form each correspond- 
ing pixel of the Grayscale Image 212. 

The color component images, that is, Grayscale Images 
212R, 212G and 212B are constructed in an analogous 
manner, but wherein each is generated using only the values 
for a single corresponding color component, that is, red, 
green or blue. 

c. Mechanisms of Find Area Program 314, Introduction 

Once a Vi-size grayscale image Grayscale Image 212 is 
available from Processing Program 312, Find Area Program 
314 provides the mechanisms for performing the operations 
necessary to generate the remaining images used in finding 
a Graphic Icon 144 region and constructing a Graphic Icon 
144. These operations include: 

(1) Generating a low-pass filtered version, Filtered Gray- 
scale Image 214 of Grayscale Image 212 to blur and 
diminish any fine text and detail, preferentially leaving 
any bold image content relatively prominent; 

(2) Generating an Edge Image 216 from Filtered Gray- 
scale Image 214, the Edge Image 216 having high pixel 
values in regions where Filtered Grayscale Image 214 
has certain kinds of structure, and an ultra-low- 
resolution Vie-size version of Edge Image 216, that is, 
Coarse Edge Image 218, for use in an initial quick 
search; o 

(3) Replacing each "pixel" of the Coarse Grayscale Image 
218 by the sum (modulo register size) of a correspond- 
ing rectangle-to-there from 0,0, wherein an arithmetic 
combination of up to four such entries yields the sum 
of the corresponding pixels of the corresponding can- 
didate rectangle of the Coarse Grayscale Image 218 and 
a value representing a "figure of merit" representing, in 
turn, the proportion of "image edge" in the candidate 
rectangle; 

(4) Searching essentially the entire Coarse Grayscale 
Image 218 to find a candidate rectangle of high "merit", 
that is, high proportion of "image edge", which will 
serve as the starting point for subsequent enlargement, 
reduction or modification of the region to be used in 
constructing a Graphic Icon 144. 

(5) Extending the left, top, right, and bottom sides of the 
candidate rectangle outward so long as enough edge 
value, as determined above, continues to accumulate so 
as to maintain the "figure of merit" of the growing 
candidate rectangle; 

(6) Adjusting the sides of the candidate rectangle for the 
purpose of (a) trimming back relatively empty side 
areas of the image and (b) extending the sides of the 
candidate rectangle outward where the rectangle side 
seems to truncate sufficiently strong structural elements 
in the image; and, 
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(7) Adjusting the candidate rectangle to obtain a rectangle 
having one of the selected aspect ratios chosen for 
efficient tiling of screen space — such as 6:2, 4:3, 3:4 
and 2:6, a final adjustment is made and presented as a 
"fattened" version of the rectangle to iconize. 
It should be noted that Find Area Program 314 has 
identified only the part of the Original Image 210 that is to 
be re -sized, and perhaps slightly re-shaped, to form the final 
icon. Visual Link Capture Engine 138 then generates the 
final Graphic Icon 144 by reference to the Original Image 
210 in its original resolution as the source from which the 
Graphic Icon 144 is generated. 

The mechanisms implemented by Find Area Program 314 
also address the possible issue that a pair of Graphic Icons 
144 from different Original Images 210 may turn out to be 
sufficiently similar as to be visually confusing. The mecha- 
nisms of Find Area Program 314 therefore execute steps 4 
through 7 above twice to obtain an alternate region of the 
Original Image 210 to use as the Graphic Icon 144. On the 
second pass through the Original Image 210, that is, through 
the grayscale representations of the Original Image 210, the 
mechanisms of Find Area Program 314 preclude the area of 
the first candidate rectangle from becoming any part of the 
initial candidate rectangle generated by step 4 although, in 
the present implementation, the second candidate rectangle 
is not precluded from growing into and including part of, or 
even all of, the first candidate rectangle. The final output of 
the Find Area Program 314 mechanisms thereby include 
four areas, the first candidate rectangle, the first candidate 
rectangle adjusted to a desirable aspect ratio, the second 
candidate rectangle, and the second candidate rectangle 
adjusted to a desirable aspect ratio. The choice between the 
two candidate rectangles may then be made manually by a 
user, for example, in a Visual Link Capture Engine 138 as 
implemented in a Local System 112, or automatically, for 
example, in a Visual Link Capture Engine 138 as imple- 
mented in a Server System 114. In the latter instance, the 
Visual Link Capture Engine 138 or, more probably, Visual 
Link Organizer 136, will retain a grayscale image from the 
newly generated Graphic Icon 144 selection and generation 
process, such as Edge Image 216 or Coarse Edge Image 218, 
and will compare the grayscale images of the two candidate 
rectangles with equivalent stored grayscale images of pre- 
viously generated Graphic Icons 144. 
d. Mechanisms of Find Area Program 314, Detailed Descrip- 
tion 

1. Lowpass Filter 324 

Referring again to FIG. 3A for a detailed description of 
the structure and operation of the mechanisms implemented 
by Find Area Program 314, it is shown therein that Visual 
Link Capture Engine 138 includes a Lowpass Filter 324 
connected from Grayscale Image Generator 316 to con- 
structs an image, that is, Filtered Grayscale Image 214, from 
each Grayscale Image 212 wherein the final details of the 
Grayscale Image 212 image are "smeared", that is, wherein 
the high spatial frequency elements are degraded or 
suppressed, while the low spatial frequency elements, that 
is, the larger image components, are preserved. In practice 
this means generating each destination pixel, that is, each 
pixel of Filtered Grayscale Image 214, from a weighted sum 
of pixels of the corresponding neighborhood of the source 
image, that is, Grayscale Image 212. The corresponding 
program code illustrates two possible smudging filters 
having, respectively, "radii" of 1 and 2 pixels, which apply 
the following weighting array to each target array of Gray- 
scale Image 212 pixels to generate a corresponding pixel of 
Filtered Grayscale Image 214: 
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It will be understood that these low-pass filters are illus- 
io trative and representative of a wide variety of sizes and 
patterns of such arrays of weightings that may be employed 
in low pass filtering of pixel image data. Many alternative 
filters will be known and understood by those of ordinary 
skill in the arts and would be appropriate and, in environ- 
15 ments with particular statistically frequent image 
characteristics, one or another alternative filter may be 
preferable. 

2. Edge Image Generator 326 

Edge Image Generator 326, referred to as "make edge 

2Q pics", is connected from Low Pass Filter 324 to receive each 
Filtered Grayscale Image 214 and to construct two "edge" 
images from Filtered Grayscale Image 214, one being the 
corresponding Edge Image 216 and the other being the 
corresponding Coarse Edge Image 218. As has been 
described, the grayscale image is Y* the linear size of the 

25 original image and the edge image is the same size, in pixels, 
as the grayscale image while the coarse edge image is V< the 
linear size of the grayscale image, thereby being Vie the 
linear size of the original image. 

For purposes of the present invention and the present 

30 description, the amount or degree of "edge" at a pixel in an 
image is the degree of coarse structure non-uniformity at 
that pixel as indicated by the rate at which the coarse 
structure brightness is changing in some direction. In the 
present implementation, Edge Image Generator 326 selects 

35 a pixel as a target pixel and examines the brightness value, 
as determined as described previously, a selected distance to 
either side, or above or below, the target pixel, such as the 
pixels located two pixels above and below the target pixel, 
and determines the brightness value of the two pixels and the 

40 absolute value of the difference between the two brightness 
values. Edge Image Generator 326 also determines the 
absolute value of the difference between the brightness 
values of the pixels the selected distance to the right and left 
of the target pixel. The sum of the absolute values of the 

45 difference in brightness values is considered the degree of 
"edge" at the target pixel. Alternatively, the maximum of 
these values may be used. 

This operation may be represented graphically as illus- 
trated in FIG. 3 B and expressed as value-(|2-up-2-down|+ 

50 |2-left-2-right|). 

The image of edge values may be still further improved by 
increasing the dynamic range, transposing the image to 
render low values invisible to further computation, and 
clipping the value to an allowed storage range. It will be 

55 realized by those of ordinary skill in the arts, that all of the 
constants involved are adjustable to yield the best results in 
a particular universe of images, the values in the present 
example having been found to be acceptable for World Wide 
Web pages as typically designed for 256-color colormapped 

60 display at 480-by-640 screen resolution. 

In an alternate implementation, and instead of the sum of 
values, Edge Image Generator 326 may be implemented to 
take the greatest of these differences, or to use diagonally 
located neighbors, or neighboring pixels at greater or lesser 

65 distances away. 

Other indications of being on or near an edge of coarse 
structure may be used instead or in addition to the above 



US 6,1* 

33 

methods. For example, whether a given target pixel is near 
by not exactly on a prominent edge or in the middle of a 
broad line may be detected by analysis of one of the pixel 
patterns represented by FIGS. 3C, 3D, 3E and 3F and their 
mirror images and 90° rotations wherein a, b, c and d are the 
brightness values of pixels 3 to the left, 1 to the left, 1 to the 
right and 3 to the right of the target pixel, and may be 
expressed for FIGS. 3C, 3D, 3E and 3F, respectively, as 
max(a,b)-min(c,d), max(a,b,c)-d, a-min(b,c,d) and max(a, 
d)-min(b,c) 

In yet other implementations, Edge Image Generator 326 
may also measure for steps up, or down, or at different 
distances or directions, using the strongest such result, or a 
combination of the results. 

In still other implementations of Edge Image Generator 
326 wherein the Visual Link Capture Engine 138 utilizes the 
color information available from the original image, and as 
described previously, Edge Image Generator 326 may gen- 
erate either a single Edge Image 216 incorporating or 
combining information regarding the three color 
components, red, green and blue, of an Original Image 210 
into a single image, or may generate separate Edge Images 
216 and Coarse Edge Images 218 for the three color com- 
ponents. It will be noted in this regard, however, that the use 
of reduced scale edge image is intended to reduce the 
volume of data that must be processed while implementa- 
tions using multiple edge images require significantly 
increased image processing with consequent increased 
requirements for processing speed and power or, 
alternatively, slower processing due to the additional data to 
be processed. In either implementation, the color informa- 
tion would be used to determine edges of image structures 
in three dimensional color space. Also as described, Edge 
Image Generator 326 may use for each entry the maximum 
or average values of the color component values, or still 
other combinations. Still other method for mapping strong 
image features will be useful for yet other particular appli- 
cations. 

Regardless of which method is used, however, to generate 
Edge Images 216 and Coarse Edge Images 218, the essential 
principle is to seek a two dimensional image with high 
values in locations corresponding to prominent structural 
features in the original image, whether the structural features 
are line art, such as in logos or large text, or pictures. In 
general, line art will have localized strong edges in the sense 
herein while pictures, such as photographs, tend to have 
continuing undulation of value and, while its edge maxima 
may be lesser than line art, the average edge value over an 
area of a picture may be equivalent to that of line art. It 
should be obvious to one versed in the arts of picture 
processing that, by adjustment of various parameters of the 
computation, one can strike a desirable balance between the 
influence of discrete-valued line art and that of bitmapped- 
continuous grayscale pictures. 

Finally, and in addition to Edge Images 216, Edge Image 
Generator 326 also, and at the same time as it generates a % 
size Edge Inage 216, generates a linearly 4 times coarser 
edge image by accumulating the values generated for Edge 
Image 216 to comprise the pixel values of the corresponding 
Coarse Edge Image 218, with the accumulated values of four 
Edge Image 216 pixels comprising a value for a correspond- 
ing pixel of Coarse Edge Image 218. It should be noted, in 
this regard, that while an Edge Image 216 is created as a set 
of byte sized values, the corresponding Coarse Edge Image 
218 is accumulated as a set of 16-bit unsigned word values. 

As also illustrated in FIG. 3A, Rectangular Sum Array 
Generator 328 operates on each Coarse Edge Image 218 to 
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generate a corresponding Rectangular Sum Array 330 
which, in turn, allows Visual Link Capture Engine 138 to 
quickly determine, or evaluate, the "edge figure of merit" of 
any sub-rectangle within Coarse Edge Image 218, that is, the 

5 degree of significant image features or structure within any 
sub-rectangle of a Coarse Edge Image 218 as indicated by 
the amount of "edge" within the sub-rectangle. It will be 
noted that any subrectangle must generally have certain 
minimum dimensions and must generally fall short of the 

io full image dimensions in each direction by at least a small 
minimum amount. 

As illustrated in FIG. 3G, a Rectangular Sum Array 330 
contains a Storage Location 332 for and corresponding to 
each "pixel" of the corresponding Coarse Edge Image 218. 

15 Rectangular Sum Array Generator 328 stores, in each Stor- 
age Location 332 in Rectangular Sum Array 330, the sum of 
the values of all pixels of Coarse Edge Image 218 contained 
within the area bounded by 0,0 as the origin at the upper left 
comer of the Rectangular Sum Array 330, which corre- 

20 sponds to the upper left corner of the corresponding Coarse 
Edge Array 218, and the location of the Coarse Edge Image 
218 pixel corresponding to the storage location in the 
Rectangular Sum Array 330. For example, in FIG. 3G 
Storage Location 332a contains the sum of the values of all 

25 pixels in the Coarse Edge Image 218 in the rectangular area 
bounded on the upper left by the origin (0,0) and on the 
lower right by the pixel corresponding to the location of 
Storage Location 332a, Storage Location 332a* contains the 
sum of the values of all pixels in the Coarse Edge Image 218 

30 in the rectangular area bounded on the upper left by the 
origin (0,0) and on the lower right by the pixel correspond- 
ing to the location of Storage Location 332d, and so on. It 
will be noted, in this regard, that the pixels corresponding to 
Storage Location 33 2 a, 332/?, 332c, and so on are located 

35 immediately outside the rectangles of interest defined by the 
origin an a corresponding one of Storage Locations 332. 

The relative value of the total amount of "edge" structure 
of features appearing in any sub-rectangle of interest in a 
Coarse Edge Image 218 can thereby be determined from 

40 sum values stored in the Rectangular Sum Array 330 as the 
value sum a+sum d-sum b-sum c wherein sum a, b, c and 
d are the stored sum values for the comers of the sub- 
rectangle of interest. To do so, Rectangular Sum Search 
Mechanism 334, described below, identifies the Storage 

45 Locations 332 corresponding to the pixels in the Coarse 
Edge Image 218 corresponding to the upper left and lower 
right comers of the sub-rectangle of interest and subtracts 
from the sum value stored therein, which is the sum of all 
Coarse Edge Image 218 pixel values in the rectangle defined 

50 by the origin and the pixel corresponding to that Storage 
Location 332, the sum values stored in Storage Locations 
332 corresponding to the upper right, lower left and upper 
left comers of the sub-rectangle of interest. These Storage 
Locations 332 corresponding to the upper right, lower left 

55 and upper left of the sub-rectangle of interest contain the 
sums of all Coarse Edge Image 218 pixel values in the 
Coarse Edge Image 218 sub -rectangles defined by the origin 
and the pixels corresponding to the upper right, lower left 
and upper left comers of the sub -rectangle of interest, and 

60 therefore contained within the array sum stored in the 
Storage Location 332 corresponding to the lower right 
comer of the sub-rectangle of interest and the result of the 
operation will be the sum of the values of the Coarse Edge 
Image 218 pixels contained within the sub-rectangle of 

65 interest. For example, and referring to FIG. 3G, the sum of 
the edge values of the .Coarse Edge Image 218 pixels in the 
sub-rectangle defamed by Storage Location 332 is deter- 
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mined by subtracting from the sum value stored therein the 
sum values stored in Storage Locations 332a, 332fc and 
332c, thereby subtracting the sum values of the three sub- 
rectangles defined by the origin and Storage Locations 332a, 
332* and 332c from the sum value stored in Storage 5 
Location 3324, thereby generating a sum value for the 
sub-rectangle defined by the origin and Storage Location 
3324. 

It will be noted that if the rectangle of interest extends 
from the left and/or the top of the image, the calculation is 10 
correspondingly simplified. That is, if any of the pixels 
corresponding to Storage Locations 332a, 3326, 332c, and 
so on are outside of, that is, above or to the left of, the 
original total image, their rectangle-sum values are not 
looked up on the sums array, but instead the corresponding 15 
value is understood to be zero. Alternatively, the entries of 
the rectangle-sums array may be stored into and fetched 
from one position to the right of and below where previously 
indicated, with the top row and left column filled with zeros. 
There are then no special cases to consider as a fetch above 20 
the top or to the left of the left edge will automatically result 
in a zero value. 

It is apparent that a sum value for any sub -rectangle of a 
Coarse Edge Image 218 can be quickly and easily deter- 
mined. It will also be apparent, as will be discussed below, 25 
that the sum values for the sub-rectangles of a given Coarse 
Edge Image 218 can thereafter be used to quickly and easily 
determine the relative figures of merit of those sub- 
rectangles wherein a figure of merit essentially indicates 
whether a sub-rectangle contains enough edge, that is, 30 
"sum", in relation to the size of the sub -rectangle. 

It should be noted that sum values may exceed the 
maximum value that may be held in a machine word. This 
is generally acceptable, however, provided that overflow- 
traps or other such cautionary hardware or software features 35 
are not activated in the system in which Visual Link Capture 
Engine 138 is executing, as the sum modulo 2-to-the-power- 
of-bits-per-word will be stored, that is, the sum starts count- 
ing over, e.g. 65534, 65535, 0, 1, 2, . . . 2. Also, in most 
computer systems, the above arithmetic d+a-b-c will be 40 
performed modulo the same number, yielding the desired 
value, provided only that the highest sum that is expected is 
lower man the modulus, in which case the capture engine as 
implemented could add results from several such sub-parts 
of the indicated rectangle so long as no part's sum exceeds 45 
the maximum possible such sum. 

Finally, the search for the optimum candidate rectangle, 
described below, can be exhaustive, and involves tests on 
thousands of possible rectangles. It will be apparent, 
however, that each test is relatively fast to perform, involv- 50 
ing calculations of only a few values each. In addition, iu the 
present implementation the conversion of a Coarse Edge 
Image 218 to a Rectangular Sum Array 330 is performed in 
a very simple and straightforward way, computing first the 
top row and the left column, then filling each remaining 55 
Storage Location 332 in the remainder of the array from the 
values immediately above, immediately to the left, and 
immediately diagonally above and to the left. 

3. Rectangular Sum Search Mechanism 334 

As illustrated in FIG. 3A, Rectangular Sum Search 60 
Mechanism 334, implemented as "find_start__rect" 
searches the Rectangular Sum Array 330 corresponding to a 
Coarse Edge Image 216 of an Original Image 210 for 
candidate regions to be potentially used in generating a 
Graphic Icon 144 and determines a "best" first approxima- 65 
tion to an acceptable Graphic Icon 144 region in the Original 
Image 210 as a Candidate Region 336A 
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In this regard, in the present implementation of Rectan- 
gular Sum Search Mechanism 334 the figure of "merit" or 
"assurance" of each candidate rectangle is determined, as 
described previously, first, on the average "edge" values of 
the pixels, that is, on the values of the pixels of Coarse Edge 
Image 218 as represented by the sum values stored in the 
corresponding Storage Locations 332 of Rectangular Sum 
Array 330, for each potential sub-rectangle of Rectangular 
Sum Array 330, and, as a secondary factor, on the size of 
each candidate rectangle, for the reason that a very small 
rectangle can easily but not meaningfully exceed larger 
rectangles as regards the average "edge" value of the pixels 
represented therein. Stated another way, a larger region or 
rectangle that includes a smaller region or rectangle of 
higher average "edge" value is probably a meaningful 
enlargement of the smaller region or rectangle. 

In an alternate embodiment of Rectangular Sum Array 
Generator 328, the figure of "merit" or "assurance" of a 
rectangle is implemented as the average edge value of the 
rectangle, as determined by the value sums stored in Rect- 
angular Sum Array 330, plus a constant times the perimeter 
of the rectangle in pixels. Clearly a variety of such alternate 
evaluations of "merit" or "assurance" might be used. The 
principle implemented in the presently preferred embodi- 
ment of Rectangular Sum Array 330, however, is that for 
rectangles having approximately equal values of "merit" or 
"assurance", the larger the rectangle the better. In the 
reverse, for rectangles of equal size, the greater the amount 
of edge (bold structure), the better. 

Having identified a "best" Candidate Region 33 6A, Rect- 
angular Sum Search Mechanism 334 in the presently pre- 
ferred embodiment of Visual Link Capture Engine 138 then 
repeats the search procedure to determine at least a second 
Candidate Region 33 6B, thereby determining and identify- 
ing at least two candidate rectangles from each Original 
Image 210, as previously described. As has been and will be 
discussed, the identification of two or more Candidate 
Regions 336 thereby allows Visual Link Capture Engine 138 
or the user to select the Candidate Region 336 most visually 
representative of the original image or to compare the 
Candidate Regions 336 to previously existing Graphic Icons 
144 and to select and retain the Candidate Region 336 most 
visually distinct from all previously existing Graphic Icons 
144 without having to repeat the entire identification process 
from the beginning. 

In the presently preferred implementation, Rectangular 
Sum Search Mechanism 334 is not permitted to retrace the 
search through the image by which Candidate Region 336A 
was identified when locating and identifying Candidate 
Region 33 6B. Rectangular Sum Search Mechanism 334 is 
thereby prevented from identifying the same or nearly the 
same region of the image as Candidate Regions 336A and 
336 B as it is desirable to locate two visually distinguished 
Candidate Regions 336. As such, and for this reason, Rect- 
angular Sum Search Mechanism 334 is designed so that 
Candidate Region 336A is designated as a forbidden area of 
the image and the search operation performed by Rectan- 
gular Sum Search Mechanism 334 is accordingly con- 
strained by the designation of Candidate Region 336A as a 
forbidden area to avoid overlap with a forbidden area if in 
fact, a Candidate Region 33 6A has been previously identi- 
fied. 

It will be appreciated by those of ordinary skill in the arts 
that there are a number of methods for constraining the 
Rectangular Sum Search Mechanism 334' s search for Can- 
didate Region 336B to prevent overlap, or at least significant 
overlap, with a Candidate Region 33 6A. For example, a 
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second rectangle will not overlap a first rectangle if and only 
if the second rectangle is completely above, or completely to 
the left of, or completely to the right of, or completely below 
the first rectangle. Rectangular Sum Search Mechanism 334 
may be implemented, however, to be constrained from 
considering any rectangular region of Rectangular Sum 
Array 330 wherein one or both of the upper left or lower 
right corners of the rectangular region fall within the rect- 
angular region of Rectangular Sum Array 330 comprising 
Candidate Region 33 6 A. However, while a constraint with 
respect to both corners will prevent any overlap between 
Candidate Regions 336A and 336B, a constraint with respect 
to only one comer would permit at least some overlap, 
unless further constraints are imposed on the possible loca- 
tions of a Candidate Region 336 B corner within Candidate 
Region 336A. 

The present implementation of Rectangular Sum Search 
Mechanism 334, is not constrained however, so that the fully 
developed Candidate Region 33 6B will not overlap the fully 
developed Candidate Region 336A, as the full extents of 
Candidate Regions 33 6 A and 336 B will not be known until 
the regions are fully developed, as described below. As such, 
an overlap may occur between the fully developed Candi- 
date Regions 336A and 336B, particularly when there is no 
acceptable but completely separate second Candidate 
Region 336B. 

Finally, it should be noted that search is performed by 
Rectangular Sum Search Mechanism 334 in what may be 
referred to as "Vie scale space", that is, within the dimen- 
sions and coordinates of Coarse Edge Image 218 and 
Rectangular Sum Array 330. s such, Rectangular Sum 
Search Mechanism 334, having identified Candidate 
Regions 33 6 A and 336B in terms of the coordinates and 
dimensions of Coarse Edge Image 218 and Rectangular Sum 
Array 330, transforms the coordinates defining the bounds 
of Candidate Regions 33 6 A and 336B, that is, the coordi- 
nates of the comers of Candidate Regions 33 6 A and 336B 
into corresponding coordinates in the " l A scale space" of 
Grayscale Image 212, Filtered Grayscale Image 214 and 
Edge Image 216 by, in effect, magnifying the coordinates of 
Candidate Regions 336A and 336B by a factor of four. The 
final results of the search are thereby reported in "Vi scale 
space" as Initial Candidate Regions 336AI and 33 6B for 
final selection and adjustment of the selected region or 
regions as Graphic Icons 144, as described below. 

4. Grow Rectangle Mechanism 338 

Grow Rectangle Mechanism 338, as implemented by 
"grow__rect", cyclically and iteratively attempts to "grow" 
each of Initial Candidate Regions 336AI and 336BI by 
extending each side of each of Initial Candidate Regions 
336AI and 336BI outward. A "figure of merit" or "assur- 
ance" for each of Initial Candidate Regions 336AI and 
336BI is evaluated after each such extension, with the 
purpose of searching for larger Candidate Region 336Aand 
336B rectangles with higher figures of "merit" or "assur- 
ance". 

In the initial steps of the "growing" process, a few 
possible extensions are considered to the right, then upward, 
then to the left, and then downward. Then, and dependent 
upon the results of the initial attempts, progressively larger 
ranges of extension are attempted and evaluated. As a result 
of this process, the extents of either or both of Initial 
Candidate Regions 336AJ or 336BI may be extended to 
"bridge over" possible "blank" areas, that is, adjacent areas 
having little or no significant visual structures or features, if 
and when there are nearby areas having significant and 
presumably related visual structures or features that may be 
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preferably incorporated into either or both of Candidate 
Regions 336A or 336B. 

As indicated in FIG. 3 A, Grow Rectangle Mechanism 338 
may perform this "growing" operation using Edge Image 

5 214 or Coarse Edge Image 216 or Rectangular Sum Array 
330. As has been described previously, both Coarse Edge 
Image 218 and Edge Image 216 contain numeric values for 
each pixel therein that represent the "edge" characteristics or 
properties of the pixels while Rectangular Sum Array 330 

10 contains an accumulated value representing the cumulative 
totals of edge information for the rectangles defined by each 
pixel in Coarse Edge Image 218. In addition, the coordinates 
and dimensions of Initial Candidate Regions 336AI and 
336BI have been determined by Search Rectangle Mecha- 

15 nism 330 in terms of the lil A scale" coordinates of Edge 
Image 216 and in terms of the "Vi6 scale" coordinates of 
Coarse Edge Image 218 and Rectangular Sum Array 330. 

The operation of Grow Rectangle Mechanism 338 will 
therefore be essentially the same whether Grow Rectangle 

20 Mechanism 338 operates in U1 A scale" space or in "Vie scale" 
space, although the present implementation of Grow Rect- 
angle Mechanism 338 operates in "Vi scale" space. In either 
implementation, Grow Rectangle Mechanism 338 will 
determine an initial figure of "merit" or "assurance" for each 

25 of Initial Candidate Regions 336AI and 336BI wherein the 
figure of "merit" or "assurance" represents the amount of 
edge information initially within the rectangles defined by 
Initial Candidate Regions 336AI and 336BI. This informa- 
tion is already available for "Vie scale" space from the 

30 operations of Search Rectangle Mechanism 330, as 
described above, and an equivalent figure of "merit" or 
"assurance" can be determined from the information stored 
in Edge Image 214 for k Va scale" space. 

Grow Rectangle Mechanism 338 will then extend each of 

35 Initial Candidate Regions 336AI and 336BI by a predeter- 
mined amount in an initial predetermined direction, such as 
one pixel to the right, and will evaluate the figure of "merit" 
or "assurance" for the resulting extended rectangles. This 
process will be repeated by successive amounts, and in 

40 different directions, for amounts of increase and directions 
for which the figure of "merit* ' or "assurance" of an extended 
rectangle remains approximately the same as the initial 
value or increases. If the figure of "merit" or "assurance" 
declines for the smaller initial extensions, the process will be 

45 repeated for greater extensions, such as two, four or eight 
pixels in each direction. 

The final result of the operations of Grow Rectangle 
Mechanism 338 are then provided as Grown Candidate 
Regions 336AG and 336BG. 

50 5. Edge Adjuster 340 

Edge Adjuster 340, implemented in Appendix A as 
"adjust_edges", operates upon Grown Candidate Regions 
336 AG and 336BG to perform a subsequent adjustment of 
the sides of Grown Candidate Regions 336AG and 336BG. 

55 As has been describe, Edge Adjuster 340 "sims away" pixels 
along the sides of the rectangles having relatively little 
visual structure or features and extends the sides of the 
rectangles where visual or structural features extend across 
the sides of the rectangles, thereby incorporating the con- 

60 tinuing visual structures or features within the rectangles, 
and generating Edge Adjusted Candidate Regions 336AE 
and 336BE. 

Edge Adjuster 340 operates according to the criteria 
described previously to cyclically and iteratively adjust the 
65 boundaries of Grown Candidate Regions 336AG and 
336BG by one -pixel adjustments, either inwards or outward- 
son each edge, to incorporate the entirety of such features as 
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pictures and logos within the rectangles, even when such 
extensions do not meet the edge information figure of 
"merit" or "assurance" criteria used by Search Rectangle 
Mechanism 330 or Grow Rectangle Mechanism 338. 

As the criteria used by Edge Adjuster 340 has been 
discussed previously, Edge Adjuster 340 will not be dis- 
cussed in further detail herein. It will be apparent to those or 
ordinary skill in the relevant arts, however, that many 
methods may be used to adjust the regions enclosed in either 
or both of Grown Candidate Regions 336AG and 336BG to 
include visual features and structures that extend outside the 
rectangles, or to eliminate peripheral areas of the rectangles 
having little visual structure or features. The essential aspect 
of the operation of Edge Adjuster 340 is to find at least one 
nucleus area containing strong image material and to adjust 
its sides, or boundaries, so as to differentiate visually objects 
contained therein from extraneous areas that ought not to be 
included. For this purpose, Edge Adjuster 340 operates to 
grow or enlarge the initial candidate area so as to include an 
agreeable relationship between coarse structure and size and 
to adjust the sides of the initial candidate area in or out to 
appropriate cut-off points with the goal of totally including 
internal visual objects while excluding nearby parts of 
external visual objects. 

It should be noted that the initial rectangles of Candidate 
Regions 336Aand336B are determined in "Vie scale" space, 
as described above, and that all subsequent operations, such 
as growing Candidate Regions 336 A and 33 6B and adjusting 
the sides of Grown Candidate Regions 336AG and 336BG 
are performed in tf Y* scale" space in the present implemen- 
tation of Visual Link Capture Engine 138. As such, the 
granularity of the grown and adjusted Candidate Regions 
336A and 33 6B will be limited to the granularity of "V£ 
scale" space. In alternate embodiments, however, and with 
additional computing time and more storage allotted to 
image arrays, Visual Link Capture Engine 138 can maintain 
and use half-size and/or full size arrays, alternatively or 
subsequently, and perform these operations in "Yi scale" or 
"full scale" space, yielding a 2 pixel or single pixel granu- 
larity for the final growing and adjustment of Candidate 
Regions 336A and 33 6B. While this increase in resolution, 
or refinement of granularity, may result in ultimately finer 
adjustments, the benefit may be marginal given that the 
location of coarse structure is the essential goal. 

6. Aspect Ratio Adjuster and Candidate Comparator 342 

Finally, Aspect Ratio Adjuster and Candidate Comparator 
342, "fix_aspect_ratio" accepts proposed source image 
areas Edge Adjusted Candidate Regions 336AE and 336BE 
and enlarges Edge Adjusted Candidate Regions 336AE and 
336BE horizontally or vertically to conform to one of a set 
of desirable aspect ratios. As described previously, in the 
present example those aspect ratios are selected as 6:2, 4:3, 
3:4 and 2:6, including a border area. 

Aspect Ratio Adjuster and Candidate Comparator 342 
also compares the ratio adjusted Edge Adjust Candidate 
Regions 336AE and 336BE, in the manner previously 
described, to select between the candidate regions, and 
compares the selected ratio adjusted candidate region with 
previously generated Graphic Icons 144, if any, to determine 
whether the selected candidate region is too similar to a 
previously existing Graphic Icon 144, reverting to the unse- 
lected candidate region if too similar a match is found with 
a previously existing Graphic Icon 144, or even reinitiating 
the generation process if necessary. As has been described 
elsewhere herein, and for other processes, the selection 
process may be performed automatically by the capture 
engine, for example, by performing an overall comparison 



51,838 Bl 

40 

between two Graphic Icons 144, or manually by displaying 
the Graphic Icons 144 to a user for selection at the user's 
decision. 

As has also been described, the adjustment is performed 

5 by means of selected intermediate ratios used as aspect ratio 
range dividers. For example, if a Candidate Region 336 
heigh th-to-width ratio is less than 4:3 and greater than 3:4, 
and taller than wide, the Candidate Region 336 is 
"stretched" vertically to an aspect ratio of 4:3. Otherwise, if 

io the ratio lies within this range the Candidate Region 336 it 
is stretched horizontally to an aspect ratio of 3:4, as the 
applicable aspect ratio range divider in this example occurs 
at 1:1. Other range dividers of the present example have 
been chosen to occur at 9:5 and 5:9. 

15 Many aspect ratios and aspect ratio range dividers are 
possible, depending upon the number and range of accept- 
able aspect ratios for Graphic Icons 144. For example, 
another set of icon aspect ratios might be 6:2, 5:3, 4:4, 3:5 
and 2:6, which would provide more advantageous "fits" for 

20 certain ranges of icons, but would result in significantly 
more icon shapes and greater consequent difficulty in orga- 
nizing the icons tightly in a display. Still other sets of 
acceptable sizes and shapes are possible, including those 
based on non-rectangular division of the ultimate display 

25 space. 

As has been described, and will be understood by those of 
ordinary skill in the relevant arts, many methods and com- 
binations of methods may be used to "stretch" or otherwise 
adjust the aspect ratios of Candidate Regions 336 to form 

30 Graphic Icons 144 or to "fit" the Candidate Regions 336 into 
the selected aspect ratios when a given Candidate Region 
336 does not fit into one of the selected aspect ratios. For 
example, any extra space around a Candidate Region 336 
after it has been fitted into a selected aspect ratio may be left 

35 blank in the display, which may result in a disorderly 
appearing display which, however, may be acceptable. In 
another implementation, the Candidate Region 336 may be 
"stretched" in the needed direction, horizontally or 
vertically, as stretches of up to 5% will scarcely be noticed 

40 and stretches of up to 10% or even more will probably not 
be objected to by a user. In yet other implementations, the 
top or bottom or sides or any combination thereof may be 
"trimmed" or extended as necessary, perhaps using one of 
the methods described herein above. In another method, the 

45 Candidate Region 336 may be extended until the region 
reaches a recognizable background color, and thereafter 
filled with that background color, rather than including 
accidental and inappropriate parts of neighboring objects. 
The operation of Aspect Ratio Adjuster and Candidate 

50 Comparator 342 has been described further herein above, 
and an implementation of Aspect Ratio Adjuster and Can- 
didate Comparator 342 as "fix__aspect_ratio" and will 
thereby be well understood by those of ordinary skill in the 
arts. As such, Aspect Ratio Adjuster and Candidate Com- 

55 parator 342 will not be discussed further herein or, and for 
the same reasons, will be the operation of various imple- 
mentations of Visual Link Capture Engine 138 in selecting 
between the two Graphic Icons 144 resulting from the 
operation of Aspect Ratio Adjuster and Candidate Compara- 

60 tor 342 on Edge Adjusted Candidate Regions 336AE and 
336 BE. In this respect, however, it will be noted that the 
object in selecting between the two Graphic Icons 144 is to 
determine whether a first one of the two Graphic Icons 144 
is sufficiently similar to a Graphic Icon 144 already in the 

65 current library or set of Graphic Icons 144 to be visually 
confusing to a user and, if so, whether the second candidate 
Graphic Icon 144 may be more suitable as being more 
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visually distinctive to a user with respect to the already manner. A further aspect of this problem in instances 

existing Graphic Icon 144. wherein there are a large number of such displays of objects 

7. Reiterative Method For Finding A "Best Rectangle" is to provide a selection of display arrangements or organi- 

It must be noted that the "best rectangle" of an original zations for each given number of objects of given sizes and 

image for use in creating a Graphic Icon 144 may also be 5 shapes to allow a variety of mutually distinctive display 

found be means of a reiterative method using essentially the arrangements for each given number of objects of given 

same mechanism and data structures as described above, but sizcs and sna P cs * 

in a reiterative manner using a plurality of progressively According to the present invention, Visual Links Orga- 
coarser edge images. Although requiring the construction of nizer 166 provides a solution to these problems by providing 
additional edge images in the same manner as described 10 a mechanism for efficient packing any arbitrary collection of 
above, this method involves simpler steps and is thereby Graphic Icons 144 having various dimensions and aspect 
faster in at least some implementations. ratios objects, or other visual display objects such as con- 
According to the alternative, recursive method, the cap- ventional icons or photographs of electronic mail recipients 
ture engine develops a quarter-size grayscale image as of various sizes and/or shapes into a display area 
described above, for example, 192 pixels wide by 144 pixels 15 It has been noted in the previous discussions of Visual 
high as these dimensions are multiples of 2, and from that Links Organizer 166 that the "tiling" of many icons of 
grayscale image develops a low-pass filtered image and a various sizes and shapes, as was illustrated in FIG. 2B, is a 
first edge image. The capture engine then generates a "bin-packing" problem of tiling geometry and that, in cer- 
sequence of progressively coarser edge images, each being tain implementations, such as those with a relatively small 
reduced in resolution by 2 with respect to the previous edge 20 number of icons per display, for example, 10 to 20, Msual 
image of the sequence. This, for example, will result in a Links Organizer 166 maybe implemented by computing and 
192x144 edge image, referred to as the first edge image, a tabulating at least one arrangement for each maximally 
96x72 second edge image, a 48x36 third edge image, a achievable icon population distribution. Thus, the possible 
24x18 fourth edge image, and a 12x9 fifth edge image. arrangements for a display or a sub-section of a display can 
The capture engine will then perform an exhaustive 25 be determined in advance, as pre-arranged packing patterns 
search as described above to find the best subarea of the fifth, for packing, for example, two 6:2s, four 4:3s, three 3:4s and 
and smallest, edge image that contains none of the fifth edge one 2:6, and stored for use in a display or a sub-section of 
image edge pixels and is at least 3 pixels wide by 2 pixels a display as necessary. 

high. The coordinates of the best rectangle of the fifth edge Therefore, and as will be described in the following 

image are then reflected back to the fourth edge image and 30 Visual Links Organizer 166 performs this function by pre- 

81 varients of that "best rectange" as reflected back to the computing and concisely storing tabulated guides or tables 

fourth edge image are tested to find the "best" rectangle for organizing visual objects, such as conventional icons, 

varient of the fourth edge image corresponding to the "best" Graphics Icons 144 or photographs of electronic mail 

rectangle of the fifth edge image. In this regard, the quantity recipients, of various sizes and/or shapes in a display space 

81 represents 3 varients on each possible edge adjustment of 35 so as to speed and simplify the solution to specific object 

the "best" rectangle from the fifth edge image, that is, with layout problems as they arise. Despite the conciseness of 

each edge being pushed out one pixel, being left alone, or these tables, however, Visual Links Organizer 166 provides 

being pulled in one pixel. a wide variety of answers to each given problem, that is, a 

The coordinates of the "best" rectange of the fourth edge variety of visual object organization layouts for each given 

image that is a varient of the £t best" rectangle of the fifth 40 number of objects of given sizes and shapes, 

edge image are then reflected into the third edge image to a. General Description of Visual Links Organizer 166 

find the corresponding rectangle therein. 81 varients of the As has been discussed with reference to FIG. 1A, Local 

"best" fourth edge image rectangle that is a varient of the System 112 includes a Visual Links Control 164 and a Visual 

"best" fifth edge image rectangle are again examined in the Links Organizer 166, both of which are associated with and 

third edge image, again by one pixel adjustments of the edge 45 operate in association with Web Access Program 162, and 

of the rectangle in the third edge image, until the "best" third associated with Visual Links Control 164 and Visual Links 

edge image rectangle is found. Organizer 166 is a Visual Links Library 168 for storing 

This processes is repeated for successive edge images up Visual Links 142, which include Graphic Icons 144. As 

through the sequence, moving to progressively higher reso- discussed, Visual Links Control 164 operates to interface 

lution edge images, until a final "best" rectangle is found. 50 and integrate Visual Links 142, Visual Links Organizer 166 

It will be appreciated by those of ordinary skill in the and Visual Link Capture Engine 138, if present, into Web 

relevant arts that this alternative method thereby achieves a Access Program 162, and allows a user to access the target 

definition of a subrectange for use in creating a Graphic Icon of a Visual Link 142 and to save a Visual Link 142 in Visual 

144 by successive approximation. It will also be appreciated Links Library 168 or a file. Visual Links Organizer 166, in 

that this method can, in principle, define any possible 55 turn, allows a user to create and edit Visual Links 142, to 

subrectangle of the largest edge image, except those con- store Visual Links 142 in Visual Links Library 168, and to 

taining the very edge row of pixels, with tests on only a few otherwise manage Visual Links 142 and Visual Links Librar- 

hundred mostly smaller rectangles. ies 168. "Visual Links Organizer 166, which will be described 

^ , ^ . , , T . , ^ . in detail further below, also allows a user to organize the 

4. Detailed Descruotion of Visual Links Organizer 6Q dispUy of yisual tj^ i4 2 in a Visual Links Library 168, 

16b (rlu. 4A-41) £ Qr exarn pi e by displaying Graphic Icons 144 in one or more 

As has been discussed briefly above, a common problem display windows or areas, 

in designing computer/user interfaces is to arrange the Referring to FIGS. 1 A and 4 A, it is illustrated therein that 

non-overlapping placement or organization of objects, such a the Visual Links Organizer 166 of a Local System 112 or, 

as conventional icons, Graphics Icons 144 or photographs of 65 for example, a Server System 114, may include a Layout 

electronic mail recipients, of various sizes and/or shapes in Generator 410, a Layout Table 412 and a Display Generator 

a display space, such as a display window, in an efficient 414, for respectively generating and storing display layouts 
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and generating displays of visual objects, depending upon residing on a Local System 112 may include Layout Gen- 

the particular implementation and function of a given spe- erator 410 to allow the user of the Local System 112 to 

cific embodiment of Visual Links Organizer 166. In many generate Plans 422 at will. In a like manner, an implemen- 

Local Systems 112, for example, as illustrated in FIG. 1A, tation of Visual links Organizer 166 residing on a Server 

the Visual Links Organizer 166 may include only a Layout 5 System 114 may include Display Generator 141, one or 

Table 412 and a Display Generator 414 and the layouts may morc u y° ut Tables 412 and ^yont Generator 410, thereby 

be provided from another system, such as another Local allowing the operators of the Server System 114 to generate 

System 112 or a Server System 114 to be stored in the ?.*P la >^8 °f Graphic Icons 144 residing in Visual Links 

Layout Table 412 for use in the Local System 112. Llbrar y 168 fo 5 usc b l thc °P cratore of Server System 114 

Id this regard, Layout Generator 410 is responsive to 10 JF mana S m S * e CTeatl0n ^ storage of V^ual Links 142 

T ♦ n « ■♦■ t ♦ ,11* u-ij- rvi therein, as well as generating Layout Tables 412 for local use 

Layout Definition Inputs 416, each including a Display m Server System 114 and po^ssibly for do wnloading to Local 

Space Definition 418a containing a specification of a s ^ [q othc 7 implementations, a system for 

selected display area, such as 12 units by 12 units, and a gencrating Uyout Tables 412 may include only Layout 

Visual Object Definition 418i> containing a specification of Generator 410 and the Layout Tables 412 generated by 

the numbers, sizes and aspect ratios of visual objects to be 15 Layout Generator 410, with the Layout Tables 412 being 

displayed therein, for generating one or more corresponding provided elsewhere, such as to a Local System 112 or a 

Layouts 420 which are encoded as Plans 424, each contain- Server System 114 for use therein. 

ing a description of a layout, or organization, of the specified It will also be understood by those of ordinary skill in the 
visual objects in the specified display space. Plans 424 are relevant arts that the applications of the various implemen- 
then stored in Layout Table 412, where they may be 20 tations and embodiments of Visual Links Organizer 166 are 
accessed by Display Generator 414 to generate a Visual not limited only to World Wide Web systems and the 
Display 426 of a specified display area containing a set of organization and display of Graphics Icons 144 but may be 
identified specific Graphic Icons 144 or other visual objects used in any application requiring the non-overlapping place- 
conforming with the corresponding Visual Object Definition ment or organization of objects, such as conventional icons 
4186 as to number, size and aspect ratio. 25 or photographs of electronic mail recipients, of various sizes 
Display Generator 414 is, in turn, responsive to Display and/or shapes in a display space, such as a display window, 
Definition Inputs 428A containing a Display Space Defied- in an efficient manner. The implementations of Visual Links 
tion 418a provided, for example, by a user or by another Organizer 166 may therefore be used, for example, to 
program operating through Visual Links Control 164 or organize and display graphic representations or icons rep- 
from a Visual Link Display Memory 428B associated with 30 resenting application programs, as in Microsoft Windows 
Visual Links Organizer 166 for storing information relating Program Manager, or to organize and display graphic rep- 
to the display capabilities of Local System 112, and speci- resentations or icons of files, as in Microsoft Windows File 
fying the display area to be used for displaying a set of Manager, or to organize and display icons or graphic 
Graphic Icons 144 or other visual objects. Display Defini- representations, such as photographs, of recipients or loca- 
tion Inputs 428A also include a set of Visual Object Iden- 35 tions in an electronic mail system, 
tifiers 430, such as pointers to specific Graphic Icons 144 b. Definitions and Terms 

residing in Visual I inks Library 168 and identifying mem- The operation of Layout Generator 410 may be illustrated 

bers of a set visual objects conforming with the correspond- with reference to FIG. 4B, which illustrates an exemplary 

ing Visual Object Definition 418a that are to be displayed in Layout 422 generated by Layout Generator 410. In this 

the defamed display area. In this regard, Visual Object 40 example, the Display Space Definition 418a specifies a 

Identifier 430 may be provided by a user of another program display area of 12 by 12 units and Visual Object Definition 

or, more probably from Visual Links Display Memory 428B 4186 has specified that the visual objects to be displayed 

because a user will generally use Visual Links Organizer 166 therein shall include three 6-high by 2-wide Graphic Icons 

to organize Visual Links 142 into groups of related Visual 144, five 4-high by 3 -wide Graphic Icons 144, two 3-high by 

Links 142 and will store the identifications of the Visual 45 4-wide Graphic Icons 144, and one 2-high by 6-wide 

Links 144 as groups in Visual Links Display Memory 428B. Graphic Icon 144, with 12 square units of unused space 

Display Generator 414 is then responsive to Display Defi- occurring in the resulting Layout 422. 

nition Inputs 428 for indexing Layout Table 412 with Before proceeding with the detailed description of Visual 

Display Definition Inputs 428, reading one or more corre- Links Organizer 116, certain of the terms and expressions 

sponding Plans 422 therefrom, and generating a Display 428 50 used in the following descriptions will be defined, and 

of the identified Graphic Icons 144 or other visual objects in illustrated by reference to FIG. 4B. These terms and expres- 

the specified display area and according to the selected Plan sions include: 

424. (a) Screen: a rectangular space allotted for tiling with 

It will be appreciated by those of ordinary skill in the icons, 

relevant arts that the component parts of a given implemen- 55 (b) Coordinates (of an icon): the coordinates of a Graphic 

tation of Visual Links Organizer 166 may depend upon the Icon 144 or other visual object in a screen is described by the 

intended purpose and functions of that implementation of coordinates stating horizontal and vertical positions, in that 

Visual Links Organizer 166. For example, and as illustrated order, of the top left corner of the visual object, using the 

in FIG. 1A, an implementation of Visual Links Organizer horizontal and vertical coordinates 0, 1, . . . ,9,A,B, . . . Z 

166 residing in a Local System 112 may include only 60 having an origin at the top left comer of the screen, as 

Display Generator 414 and one or more Layout Tables 412 illustrated in FIG. 4B. 

as the Layout Tables 412 may be provided from an external (c) Units: the basic dimensional granularity of visual 

source, such as downloaded from a Server System 114 or a object, such as Graphic Icons 144, and screen layouts. 

Web Site 118 or otherwise provided with Visual Links (d) Icons: generic term for Graphic Icons 144 or other 

Organizer 166 on initial installation of Visual Links Orga- 65 visual objects forming the tiles for filling space, illustrated 

nizer 166 or later loaded from disk. In other instances, in FIG. 4B for Graphic Icons 144 having aspect ratios of 

however, an implementation of Visual Links Organizer 166 6x2, 4x3, 3x4, and 2x6 units; 
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(e) Layout: a specific geometric configuration of icons in 
a screen, referred to herein as a Layout 422. 

(f) Orientations, rotations and reflections: an angular 
position, orientation rotation in a screen with respect to the 
horizontal and vertical axes of the screen and labeled with 
respect to the 1:00 o'clock position, using a clock face as a 
coordinate system according to the convention 

B 1 
A 2 

• 

8 4 
7 5 

so that orientation "8" specifies a reflection through a 45 
degree backslash angle. 

(g) Icon set: a specified group of icons having defined 
aspect ratios, for example, the set illustrated in FIG. 4B is 
{6x2, 4x3, 3x4, 2x6} and an alternative set might be {6x2, 
5x3, 4x4, 3x5, 2x6}. 

(h) Fault: a cutline across a layout which divides it into 
two rectangles without intersecting any icons. 

(i) Sub rectangle: a proper subarea of a layout containing 
completely some subset of its icons and no parts of other 
icons. 

(j) Subsquare: a square subrectangle. 

(k) Types: the sizes and shapes of the icon set as identified 
by the heights of the icons in the set, so that the types of the 
icons illustrated in FIG. 4B are 6, 4, 3, 2. 

(I) Population (of a layout): the number of icons of various 
sizes/shapes as specified by the number of each type stated 
in an understood order, such as decreasing height, so that the 
population of the layout illustrated in FIG. 4B is specified as 
3521, representing 2 6x2 icons, 1 4x3 icons, 2 3x4 icons, 
and 1 2x6 icon, wherein it us understood that a population 
may be hypothetical and may, in fact, be unrealizable in a 
certain size of screen, so that the population 6000 cannot be 
realized by a layout for an 11x11 screen even though there 
would seem to be sufficient space for the icons in the screen. 

(m) Subpopulation (of a population): a set of icons for 
which the count for each type therein is not greater than the 
count for corresponding type in the population of which the 
subpopulation is a member, and which is lesser for at least 
one type, so that the set 2421 is a subpopulation of the 
population 3521. 

(n) Maximal population: for a specific screen size and 
shape, a layout-realizable pupulation such that the increase 
by one of any of its icons types yields an unrealizable 
population for that screen size and shape. 

(o) Plan: a notation, referred to herein as a Plan 424, for 
a layout for understood screen size/shape, such as the 12x12 
screen illustrated in FIG. 4B and wherein the population and 
sequence of top left coordinate locations of the icons are 
ordered by type, so that the plan for the example of FIG. 4B 
is written as 

3521 00 20 0 6, 90 94 36 66 98, 40 43, 2A, 

with the understanding the commas and spaces are not 
required, wherein the first number, which contains a digit for 
each type of icon, such as 3521, defines the population of the 
screen by number of each type of icon, such as 3 6x2 icons, 
£ 4x3 icons, 2 3x4 icons, and 1 2x6 icon in the present 
example, and each set of two digit numbers defines, for each 
type of icon, the coordinates of the upper left comers of the 
icons of the corresponding type, so that 00 20 06 specifies 
the coordinates of the upper left comers of the 6x2 icons, and 
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so on. Further examples of a tabulated set of layouts is 
illustrated in Appendix B. 

(p) Inversion (of a screen's aspect ratio): a reinterpretation 
of a plan, such as 6x7, to its inverted aspect ratio, such as 

5 7x6, by 45-degree-backslash flip and performed by 
exchanging the interpretation of its x and y dimensions and 
interchanging all designations of aspect ratios stated for the 
original screen, such as 6x2<-->2x6 and 3x4<~>4x3, so that 
the two plans are inversions of each other, as illustrated in 

io FIGS. 4C and 4D. 

According to the present invention, it is necessary to 
generate and save only plans for wider-than-high and square 
screens for icon sets that contain all icons in both orienta- 
tions as the corresponding higher-than-wide, or wider-than- 

15 high, screens and the inversions of square screens may be 
generated by inversion of the generated screens. 

(q) Sublayout: a subset of a layout wherein the population 
of the subset is a subpopulation of the layout so that the 
layout may contain the population of the sublayout by 

20 occupying a part of the icon locations in the layout with the 
icons of the population of the sublayout. For example, the 
population 3421 is a subpopulation of the population 3521 
and can be arranged in the 3521 plan by using any four of 
the five locations designated for 4x3 icons in the 3521 plan. 

25 This principle is related to but different from the principle 
that a set of one or more different plans may often be 
generated from an original plan having space not occupied 
by icons by vertical or horizontal lateral movement, that is, 
sliding, of one or more of the icons of the original plan. 

30 Therefore, by visual examination of FIG. 4A it can be seen 
that at least four additional 3521 layouts may be generated 
from the 11x12 layout illustrated therein by horizontal 
movement, or sliding, or an icon and that more may be 
generated by horizontal movement of combinations of two 

35 or more of the icons therein. 

d. The Generation of Plans by A Layout Generator 410 

Referring to FIG. 4E-1, therein is illustrated the process 
executed by Layout Generator 410 in generating at least one 
Layout 422 and a corresponding Plan 424 for each specified 

40 screen, such as 6x6, 6x7, 6x8, . . . 7x7, 7x8, . . . 10x6, . . . 
11x12, .. , and so on. The following will assume that, in the 
presently preferred embodiment of Layout Generator 410, 
only one Plan 424 is stored for each population of Graphic 
Icons 144 or other visual objects to be displayed therein, 

45 although it will be understood that in other implementations 
a plurality of Plans 422 may be generated and stored for each 
combination of a screen size and aspect ratio and population 
of Graphic Icons 144 or other visual objects to be displayed 
therein. It will be understood by those of ordinary skill in the 

50 relevant arts that the following process and mechanism may 
be applied to the arrangement or layout of any form of 
visually displayable objects, such as conventional icons, 
graphic images of all forms and images such as photographs, 
in any form of display and is thereby not limited in use only 

55 to the layout of Graphic Icons 144. As such, it will be 
understood that in the following the term "graphic icon" is 
intended to refer to any form of visually displayable object 
occupying a rectangular space. 

As illustrated in FIG. 4E, therefore, the process executed 

60 by Layout Generator 410 is generating Layouts 420 and 
Plans 422 is performed by a Layout Generator 410 mecha- 
nism including a Possible Layout Generator 429. Possible 
Layout Generator 429 generates, for each screen size and 
aspect ratio and for each maximal population of Graphic 

65 Icons 144 or other visual objects, as specified by the 
numbers and types of visual objects, as, for example, by a 
Display Space Definition 416 and a Visual Object Definition 
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420, generate all possible Layouts 420 for a combination of 
screen and population by trying all possible locations of the 
various types of icons of the specified population in the 
specified screen, disallowing icon overlaps, and consider as 
candidates the endpoint Layouts 420 that can accommodate 5 
no further icons, that is, are filled by the specific population. 

As indicated in FIG. 4E-1, Possible Layout Generator 429 
includes a Search Forward Mechanism 431a which searches 
forward on each placement of each Graphic Icon 144 or 
other visual object of the current population in the current 10 
screen until possibilities have been considered, wherein a 
search forward is an attempt to place a next icon from a list 
of icons of the current population into the current screen. 

Possible Layout Generator 429 further includes a Back- 
track Mechanism 4316 connected from and interoperating 15 
with Search Forward Mechanism 431a which backtracks on 
each placement of each Graphic Icon 144 or other visual 
object of the current population in the current screen until all 
possibilities have been considered, wherein a backtrack is 
the removal of the most recently placed icon and an attempt 20 
to place a next icon from the list of icons of the current 
population into the screen and, if no subsequent icon can be 
placed after removal of the most recent icon, a removal of 
the next previously placed icon and an attempt to place a 
next icon from the list of icons of the current population. 25 

A Layouts Storage Manager 432 then stores each Layout 
422, that is, Plan 424, in a Layout Generation Memory 434, 
shown in FIG. 4A, for subsequent testing and selection. In 
this regard, a Layout Test Mechanism 436 is connected from 
Layout Generation Memory 434 through Layouts Storage 30 
Manager 432 to read each Layout 422 from Layout Gen- 
eration Memory 434 and to test each new resulting Layout 
422 against all other Layouts 420 previously generated in 
Step 430 and stored in Layout Generation Memory 434. 

Layout Previous/New Selector 438 is connected from 35 
Layout Test Mechanism 436 and selects between a new 
Layout 422 and a previously existing Layout 422 according 
to a predetermined criteria. For example, and in the present 
embodiment, if a new Layout 422 has a subpopulation of a 
previous Layout 422, Layout Previous/New Selector 438 40 
discards the new Layout 422. If a previous Layout 422 has 
a subpopulation of a new Layout 422, Layout Previous/New 
Selector 438 replaces the previous Layout 422, unless dis- 
qualified in a following step, as described below. If a 
previous Layout 422 has a subpopulation of the new Layout 45 
422, Layout/Previous Selector 438 discards the new Layout 
422 and, if a previously existing Layout 422 has the same 
population as a new Layout 422, Layout Test Mechanism 
436 selects between the previous and new Layouts 420 by 
discarding one or the other by either (1) random selection 50 
between the previous and new Layouts 420 having the same 
population or (2) selecting the Layout 422 having the greater 
asymmetry as potentially generating the larger number of 
different Layouts 420 through a subsequent step of usage- 
phase modification, described below. 55 

At this stage in the process there may still survive Layouts 
420 having populations that are subpopulations of other 
surviving Layouts 420, such as a Layout 422 that eliminated 
a previously generated Layout 422, that is, a Layout 422 
with a subpopulation of its own, that may in turn might also 60 
disqualify others Layouts 420 previously generated which 
also have subpopulations of that Layout 422 's population. A 
final selection of Layouts 420 may therefore be performed 
by a Layouts Comparator 440 wherein the populations of 
each pair of surviving Layouts 420 are compared and 65 
Layouts 420 having populations that are subpopulations of 
Layouts 420 are discarded. Alternately, this process may be 
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performed by comparison of each pair of Layouts 420 in 
Layout Test Mechanism 436, depending upon which imple- 
mentation proves most efficient in a given embodiment of 
Layout Generator 410. 

Finally, the surviving Plans 422 encoding the final sur- 
viving Layouts 420 are tabulated and stored in are tabulated, 
as demonstrated in one or more Layout Tables 412. In this 
regard, it should be noted that in a presently preferred 
embodiment of the invention Plans 422 are grouped in 
Layout Tables 412 by screen size and aspect ratio, often with 
each Layout Table 412 continuing the Plans 422 for a given 
screen size and aspect ratio, and that Layout Tables 412 may 
then be indexed, first, by available display, or screen, size 
and aspect ratio, with the selected Layout Table 412 then 
being indexed by the icon population to read therefrom a 
Plan 424, if available therein. In other implementations, the 
Plans 422 may be grouped first by populations and then by 
screen size and aspect ratio. 

It should be noted that in as much as it is often necessary 
to generate no more than one, two or three Layouts 420 for 
each maximal achievable population for a particular screen, 
Step 430 may be implemented alternatively by first search- 
ing for the full area occupancy populations for the screen for 
each possible maximal set of icons that would fill the screen 
area, with Layout Generator 410 searching for a fitting of 
each of the sets of icon shapes into the screen area. Then, for 
each population that cannot be realized, Layout Generator 
310 searches for Layouts 420 for each possible subpopula- 
tion of that population that contains one less member, 
repeating the search with successive subpopulations that are 
each reduced a different icon type and, in each branch of the 
search, repeating the search with progressively reduced 
subpopulations until a subpopulation is found that can be 
fitted into the screen area, to find all possible subpopulations 
of the original population that can be fitted into the screen 
area. The resulting Layouts 420 are then culled by compari- 
son to previously generated Layouts 420, as described 
above. 

It will be recognized by those of skill in the relevant arts 
that the procedure of Steps 430 through 440 described above 
may require significant computing power to time to com- 
plete due to the large number of possible screen sizes and 
possible plans that must be searched for each possible 
population. The method of the present invention, however, 
and as described below, provides additional initial steps to 
increase the speed and efficiency of the generation of Plans 
422. 

As illustrated in FIG. 4E-2, therefore, and according to the 
present invention, mechanism described above with regard 
to FIG. 4E-1 may further include mechanism for performing 
the following operations in association with or preceding the 
operations performed in Possible Layout Generator 429 to 
generate patterns of various instances of icon layouts in 
various screens and to provide the resulting patterns for use 
the process described above. 

In this regard, and referring to FIG. 4E-2, Layout Gen- 
erator 410 may further include a Screen Map Generator 442 
for generating, for each screen size and aspect ratio up to a 
maximum screen height (or width) of X units, a correspond- 
ing Screen Map 444 at a resolution, for example, of Y 
bits-per-unit-square in as many X bit words as the screen is 
wide (or high) wherein X is selected as a convenient word 
length for processing in the system in which Layout Gen- 
erator 410 is implemented and Y is, for example, the 
smallest dimension, in bits, of any icon. For example, many 
computers operate efficiently with 16 bit words and, accord- 
ingly X is selected as 16 in a presently preferred embodi- 
ment of Layout Generator 410. 
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The Screen Maps 444 are then provided to an Instance divisions. The nine sets correspond to Spatial Divisions 

Patterns Generator 446 that, for each icon type, that is, icon 458a, 4586, 458c and 458a*, which correspond to the four 

size and aspect ratio, generates as many screen -size comer quadrants, Spatial Divisions 458e, 458/, 458g and 

"Instance" Patterns 448 as there are possible locations in the 458A, which correspond to the halves of the two axes, and 

Screen Map 444 for that type of icon, using special encoding 5 Spatial Division 458*, which corresponds to the intersection 

for clear space at top and bottom between the placement of point of the two axes. 

a icon and a side or the top or bottom of the Screen Map 444 Thereafter, a Location Instances Sorter 460 sorts, for and 

as necessary. This process of determining whether a given according to each type of icon, the location instances of the 

instance of an icon of a given type into the Screen Map 444 icon in the screen according to their positions with regard to 

will fit, and placing or removing it, is thereby comprised of 10 nine spatial divisions of the screen, 

fast logical AND, OR and EXCLUSIVE-OR operations, Then, and for each of the nine sets, a Location Instance 

respectively. Search and Compose Mechanism 462 searches for and 

It will be noted that with regard to allowing clear space composes compatible, that is, non-overlapping, combina- 

along any side of the Screen Map 444 that if there is no icon tions of icon location instances, for example, by operation of 

type under consideration with minimum dimension less than 15 Layout Generator 410 or by presenting possible choices to 

two, then it is unnecessary to consider instances, or place- a human operator, and a following Super-object Generator 

ments of the icon in the Screen Map 444, having one-unit 466 designates each compatible combination as a Supericon 

space between the icon and an edge, unless it is necessary 464, with the purpose of filling the screen as completely as 

due to the shapes of an icon and the screen, as in the possible with Supericons 464. 

placement of a 2x6 icon on a 7x7 screen. Therefore, in 20 For each of the nine sets of icons, and in principle, all 

making instance patterns of icon positions, Step 446 omits combinations of 0, 1, 2, or 3 or more icons may be 

near-but-not-on-side locations except where necessary. considered as Supericons 464. The search for such combi- 

The result of these operations is thereby at least one nations may be restricted, and the search accordingly 

Instance Pattern 450, an example of which is illustrated in shortened, according to certain principles. 

FIG. 4F, containing a sufficiently complete set of Locations 25 First, for Spatial Divisions, or sets, 458a, 4586, 458c and 

450 for each icon type in the screen represented by Screen 458*/, which occupy the four comer quadrants of the screen, 

Map 444, each Location 448 representing a possible all combinations having at least one quadrant with zero 

instance of the icon type in the given screen. The example members may be discarded as no maximal population can 

of an Instance Pattern 450 illustrated in FIG. 4F shows a result from a combination having an empty comer quadrant, 

sufficiently complete set of Locations 450 for 3x4 icons in 30 Further, in any pattern in which a comer unit of a quadrant 

a 7x9 screen wherein each Location 448, each of which is is not occupied, one of the icons near the comer may be 

indicated by a *, identifies the location of the upper left moved, or translated horizontally or vertically, into the 

corner of a 3x4 icon for a possible placement, or instance, empty comer quadrant, and that pattern will result at least 

of an icon of that type in the given screen. one other point in the search. That is, the two instances, with 

Finally, a Location Instances Enumerator 452 enumerate 35 the icon near the comer and with the icon in the comer, are 

the instances of the icon of the given type in the screen in essentially equivalent so that the icon may as well be placed 

order of proximity to the screen corners for each screen and in the comer rather than considering the two instances 

icon type to be considered in a corresponding Instance separately. 

Enumeration 454, and provide the Instance Enumerations Second, there will most probably be one or more combi- 

454 for use by Possible Layout Generator 429. 40 nations wherein there are "slits" one unit wide between two 

As has been discussed above, the process executed by icons which cannot be filled by vertical or horizontal trans- 
Possible Layout Generator 429 involves a significant num- lation of one of the icons, and which may in fact impede the 
ber of reiterations of icon placement steps and backtrack placement of other icons. Also, the configuration that would 
steps, which in turn entails the inadvertent rediscovery of result from moving the icons to close the "slits" will most 
members of sets of icons far down through the process steps 45 probably occur elsewhere in the search and, as a result, as no 
that conflict with one another. As such, the processes and maximal population can result from a combination gener- 
operations described above may be further enhanced for the ated by vertical or horizontal translation of an icon to 
case of large screens, such as 12x12 or larger, by a method eliminate a "slit" space between two icons. Such combina- 
wherein variously located instances of each shape of icon to tions are therefore preferably discarded from consideration, 
be placed in a screen are sorted into disjoint sets according 50 It will be noted that the above two situations are instances 
to their locations in the screen and each set is treated as a wherein there are two equivalent combinations, one where 
"supericon", the supericons then being used in the screen there is a non-fillable slit and one where an icon has been 
filling process described above. moved to fill the slit and results in an equivalent non-fillable 

As illustrated in FIG. 4E-3, Layout Generator 430 may slit. In these instances it is not necessary to consider both of 

thereby further include a Screen Spatial Division Generator 55 the equivalent combinations, but only those combinations 

456 that determines, for a large screen such as 12x12 or wherein moving an icon results in the locking out of an icon 

larger, the various possible location instances of each type of placement possibility. For example, and referring to FIG. 

icon with respect to nine spatial divisions of the screen, 4H, if the 4x3 icon at the upper left were one unit further 

which are illustrated in FIG. 4G for a 12x12 screen. As may right, no new combinations or placements would result than 

be seen therein, the screen is divided into four quadrants by 60 what is possible with the icon in its present placement, so it 

horizontal and vertical axes of symmetry and four of the is not necessary to consider the combination with the icon 

spatial divisions are formed by the four corner quadrants moved one unit to the right. In the instance wherein the 

formed by the two axes of symmetry while the right and left upper left 6x2 icon and the 4x3 icon are reversed, however, 

halves of and along the horizontal axis of symmetry, the with a unit space between the icons, a movement of the 6x2 

upper and lower halves of and along the vertical axis of 65 icon to the left precludes a 3x4 icon immediately under the 

symmetry and the intersection point of the axes of symmetry 4x3 from being placed, so that this instance should be 

in the center of the screen form the remaining five spatial considered. 
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Finally, some combinations occupy exactly the same 
areas and have exactly the same subpopulations as other 
combinations and are therefore equivalent and may be 
discarded from consideration. 

The search for such combinations may be restricted and 5 
the culling of combinations according to the above prin- 
ciples may be implemented as by Supericon Designator 466 
operating in association with Location Instance Search and 
Compose Mechanism 448, or may be performed manually 
by presenting possible combinations to a human operator for 10 
decision. It should be noted that experimentation has shown 
that the application of these principles will result in approxi- 
mately 3 A of the possible combinations surviving the culling 
process, so that the generation and search process can 
accordingly be reduced to approximately % to the ninth 15 
power of the original time required, that is, to approximately 
less than a tenth of the original time required to generate and 
search layouts for large screens. 

Lastly with regard to the use of Supericons 464 to reduce 
the number of reiterations of icon placement steps and 20 
backtrack steps, it will be apparent from the above discus- 
sions that certain sets of combinations generated as 
described above will result in sets of combinations that are 
rotational or reflectional equivalents to one another. For 
example, each of several starting points of a search for 25 
combinations may start with a set of combinations with one 
combination from each of the corner quadrants of the screen, 
that is, a 458a, 458b, 458c and 458d combination wherein 
each of the combinations, when appropriately rotated or 
reflected, is equivalent to the other combinations. This may 30 
be illustrated as in FIGS. 4H and 41 wherein it is apparent 
that the two starting sets of combinations are equivalent 
because the subpopulation and covered area pattern of FIG. 
41 is the same as that of FIG. 4H when subjected to a rotation 
about the right half horizontal axis. 35 

As also discussed above, it is anticipated that the final 
usage of Layouts 420 to generate screen displays anticipates 
that at least certain Layouts 420 will be rotated or reflected 
to generate yet other Layouts 420, so that the end result of 
the entire process could be the generation of equivalent 40 
Layouts 420. 

As a consequence, the search for such combinations may 
further be restricted and the generation of combinations 
enhanced by a process wherein the starting combinations are 
pruned, or culled, or any set of starting quadruple combi- 45 
nations that are rotational or reflective duplicates of already 
generated starting combination quadruples. 

The speed and efficiency of Possible Layout Generator 
429 and thus of the subsequent processes described above 
may also be increased by further mechanism associated with 50 
Possible Layout Generator 429 or a subsequent process 
wherein a large screen is generated as a combination of two 
or more smaller screens. 

Therefore, and as illustrated in FIG. 4E-4, Layout Gen- 
erator 410 may further include a Comer Area Examiner 470 55 
connected from and to Possible Layout Generator 429 
which, for very large screens, examines the comer areas of 
each Layout 422 as it is generated and retain only Layouts 
420 whose comers are filled in unique ways, that is, discard 
one or any pair of Layouts 420 if the patterns of similar icons 60 
filling their comers are rotational or reflectional equivalents, 
that is, if one is a 5, 7, or B reorientation of the other. Comer 
Area Examiner 470 may also apply the criteria that all four 
comers of a Layout 422 be occupied. 

Finally, the speed and efficiency of Layouts Generator 430 65 
and thus of the subsequent processes described above may 
also be increased by additional processes primarily intended 
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for use with large screens, for example, greater than 12x12. 
In these additional processes, Layouts 420 are composed by 
dividing the screen into two or more smaller screens, gen- 
erating solutions for the smaller screens, and generating the 
solution for the original large screen by combining the 
solutions for the smaller screens. In this embodiment, as 
illustrated in FIG. 4E-5 and primarily for use with large 
screens, Layout Generator 410 will further include a Screen 
Sub-Area Divider 472 connected from Possible Layout 
Generator 429 which divides each screen, such as a 13x12 
screen, into two or more smaller screens or sub -areas, for 
example, (n)xl2 and (13-n)xl2 wherein n«5, 6, 7. Screen 
Sub-Area Divider 472 then provides the smaller sub-area 
screens to Possible Layout Generator 429, which uses each 
sub-area screen independently as a display area for the 
generation of corresponding Layouts 420. 

Then, a Screen Sub-Area Combiner 474 operates in 
conjunction with Layouts Comparator 440 to compose Lay- 
outs 420 for the original full size screen by joining the 
Layouts 420 generated for the smaller sub-area screens, 
adding the populations of the smaller screens, transposing 
the coordinates of the smaller screens into coordinates of the 
large, combined screen and test for duplication and inclusion 
relationships among the resulting Layouts 420, as described 
previously. 

Alternatively, the Layouts 420 may be generated for the 
smaller screens as described above and a Layout 422 may be 
generated for the original larger screen by Location Instance 
Search and Compose Mechanism 462 wherein Layout Gen- 
erator 410 or another program searches Layout Tables 412 
for a combination of smaller screens that may be combined 
into a screen having the dimensions and population of the 
large screen, and so combines the smaller screen Layouts 
420 into a Layout 422 for the large screen. 

The above described process of generating a layout for a 
large screen as a combination of layouts for smaller screens 
may be illustrated with the aid of FIG. 4J wherein an MxN 
Screen 476 is divided two SubScreens 476A and 476 B 
having dimensions, respectively, of MxA and MxB and 
which may be side by side, as illustrated in FIG. 4J, or one 
above the other. Layouts 420 for SubScreens 476Aand 476B 
that are useful in generating a Layout 422 for Screen 476 
may be generated and tabulated in advance by considering 
the generating all pairs of sublayouts for screens having 
dimensions of MxA and MxB wherein A+B=N. Layout 
Generator 410 can generate a very large number of popu- 
lations that can be arranged in MxN screens by adding 
various A+B populations. As described above, Layout Gen- 
erator 410 may discard any resulting population that is a 
subpopulation or equivalent population to one already 
generated, and may disqualify those layouts whose popula- 
tions are subpopulations of a new layout. 

It will be apparent from the above discussions that the 
method of constructing Layouts 420 for screens as combi- 
nations of Layouts 420 for smaller screens that can be 
combined into the larger screen can be extended to combi- 
nations of greater numbers of smaller screens, as illustrated 
in FIGS. 4K, 4L and 4M, which respectively illustrate a 
Layout 422 for a 7x7 screen as a combination of Layouts 
420 for four 4x3 and 3x4 screens, a Layout 422 for a 14x14 
screen as a combination of Layouts 420 for 16 4x3 and 3x4 
screens, and a 14x14 screen as a different combination of 
Layouts 420 for 16 4x3 and 3x4 screens. It is also shown in 
FIGS. 4K, 4L and 4M that the resulting screen need not have 
"faults" or lines of rotational or reflective symmetry among 
the smaller screens comprising the larger screen. 

Finally in this respect, the Plans 422 for composite layouts 
for larger screens comprised of two or more smaller screens 
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may be tabulated into Layout Tables 412 by listing the Plans It will be appreciated by those or ordinary skill in the 

422 by its subparts, either by assigned serial numbers or by relevant arts that these processes are easily programmed so 

populations, rather than by the member icons, and that the as to happen automatically, and require little time to execute, 

definitions of the composite Plans 422 can cascade. It will also be apparent that it is useful to perform these 

Also, the previously discussed methods for generating 5 operations in various orders and iteratively in various 

additional Layouts 420 from a Plan 424 stored in a Layout combinations, so long as new geometric layouts result. 
Table 412 by rotation or reflection of the stored Plan 424, 

that is, its corresponding Layout 422, may be used to 5 * Description of Visual Links 142 (FIGS. 5A-5G) 

generate additional composite Layouts 420 by rotation or As previously discussed, for example, with reference to 

reflection of the stored composite Plan 424. 10 FIG. IB, a Visual Link 142 may generally include a Uni- 

e. Using Plans 422 versal Resource Locator (URL) 128, a Tide 130, a Text 132, 

As discussed above, a user of a Visual Links Organizer and a Date 134, and may include Keywords 136, and will 

166, and in particular Display Generator 141 and one or include a Graphic Icon 144. As has been described in detail 

more Layout Tables 412, will generate requirements such as above, a Graphic Icon 144 is generated or extracted from the 

"for a 12x10 screen, provide a plan for placing population 15 graphics and text information present in the corresponding 

3321 Such a need is satisfied (or not) by a linear scan of HTML document and which serves as a visual representa- 

the appropriate Layout Table 412 for the first Plan 424 that tion to the user of the associated document or file. The role 

contains at least the desired population. The result of the 0 f Graphic Icons 144 may also be fulfilled by graphic 

search is either a Plan 424 or a failure, the latter meaning that representations of various forms of files, locations or recipi- 

the stated population of icons cannot be arranged on speci- 20 ents on a network, such as photographs of electronic mail 

fied screen, although it should be noted that if the specified recipients. 

screen is inverted with respect to screen for which a Plan 424 Iq addition to representing a document, file, location or 

exists, the request may be satisfied by inverting the Plan 424 recipient, for example, in a visual index of documents, files, 

for the screen for which a Plan 424 is found. locations or recipient, a Graphic Icon 144 is a computer 

As described above, a Plan 424 is a specific geometric 25 programming construct which permits a small bitmapped 

layout for the stated numbers of icons of the specified types p i ct ure, that is, a Graphic Icon 144, to serve as an active 

and for the specified screen and the user may use the Plan button on me of a World wide Wcb browser, elec- 

424 unaltered or, for the sake of variety in the appearances mail program or other application program. A user 

of numerous similar layouts, may perform various alter- may men initiate access t0 the document, file, location or 

ations to the Plan 424 to vary the visual appearance of the 30 recipient for a corresponding purpose by "clicking" on the 

Plan 424. button. As has been described, Graphic Icons 144 enable 

For example, catena for the search for a Plan 424 is uscrs to morc rca dily and easily recognize and remember a 

satisfied if the search population is a subpopulation of an Web site? document) file , location or recipient, by providing 

existing Plan 424. Therefore, and if the user does not have a represe ntative visual representation of the Web site, 

additional icons of the number and types necessary to fill the 35 document, file, location or recipient, than do text indicators 

available space in the Layout 422 represented by the Plan or> for example, ASCII string universal resource locators 

424, there is at least some flexibility as to which of the icon (URLs) 

positions to use and which to leave blan£ thereby varying M iUustrated fa pjQ SA , a Visuil Liok 142 concatenates, 

the visual appearance of the Layout 422 The choice of combin or otherwise associates, a normally encoded 

which icons positions to fill may be made, for example, at 40 ... \ • . * u * • w \ *aa * r 

. * , e , . bitmapped picture, that is, a Graphic Icon 144, with a set of 

the mdividual discretion of the user, for example, using u ^ ^ (c{cwi tQ ^ a ^ 

drag and drop or point and cliclr operations to associate ^ . . , nA j • i j c • ** *u 

& _ . r . v . A . r . , . Dataset 500, and includes a means for associating the 

icons with locations, or the locations may be assigned by Graphic Icon 144 and the Visual Link Dataset 500 The 

means of a random number generator, perhaps, in the ^ describe each of , hese elemen|s fa , um 

instance of reuse of the same Layout 422, using the assign- 45 ° . 

ments for an earlier use of the Layout 422 as a seed for the As . has bcen described the active button bitmapped pic 

later uses of the Uyout 422. mre 15 comprised of a Graphic Icon 144, which is, for 

In further example, the visual appearance of a Uyout 422 exlm f le - on u ,he orde f ° f . 100 b y "» Pixels in size and is 
may be varied by rotating or reflecting the Layout 422 about, ^tended to be encoded m any of the commonly known 
for example, the vertical or horizontal axis or the 1, 5, 7, B 50 means , of b J^ P J^ od ?t- ^ bUt ™ ° 0t 
orientations of the Plan 424, either at random or by user hmitc 1 d to > JPEG - G } F ™ d Microsoft BMP. The image data 
selection. Also, if the specified screen is square and the ma y be compressed by any of the methods thereby sup- 
population is of the form abba, the Layout 422 may be P^? 1 m f ach «f as used for the purposes and uses 
reoriented about the 2, 4, 8, and A axes, with reinterpreta- °{ Y 1 ^ Link M2> ,he basic structure and detailed content 
lions of the icon types, such as 6-higb<-->2-high and 55 of the picture file remains unaltered. 
4-high<~>3-high. A Visual Link Dataset 500, in turn, is an ASCII file or the 

In a like manner, if the Layout 422 contains a vertical or equivalent, such as an encrypted alphanumeric file, which 

horizontal fault, the placement of the subrectangles defined contains, most essentially, a URL 128 which, in turn, is an 

by the fault may be exchanged or, if there exists s subrect- Ascn character string, for example, 

angle in Layout 422, the area of the subrectangle may be 60 http://www.netwavelink.com/pointreview.HTML. A 

reoriented by along the 5, 7, or B axes or, if there is a minimum Graphic Icon 144 dataset for this example 

subsquare and its subpopulation is of the form abba, then might therefore appear as illustrated in FIG. SB 

the icons of the subpopulation of the subsquare may be wherein it is shown that this minimal Visual Link 

reoriented in a similar manner. Dataset 500 includes three Designators 502, respec- 

Finally, if there is empty space along any side of an icon, 65 tively indicated as Graphic Icon 144=, URL 128« 

the icon may be moved vertically or horizontally to occupy http://www.netwavelink.com/pointreview.HTML, and 

the empty space. END 504= 
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Each Designator 502 is a body or item of data related to 
the Visual Link Dataset 500 and indicates the nature of the 
following ASCII data, if any, and ends with the '=' character. 
'URL 128' of the corresponding Designator 502 illustrated 
in FIG. 5B thereby represents the URL 128 of a correspond- 
ing document, site, file, location or recipient or the like, with 
the indicating the end of the designator. 

While the URL 128 Designator 502 is the crucial item for 
contacting a web site, the user will often find it helpful if 
additional relevant information is stored in the Visual Link 
Dataset 500, thus making information available immediately 
when and if the Graphic Icon 144 is stored locally. A typical 
Visual Link Dataset 500 with such additional information 
may appear as represented in FIG. 5C wherein the Visual 
Link Dataset 500 is illustrated as including the Designators 
500 represented as Graphic Icon 144=, URL 128=, Text 132 
including a Subject 506=and an Author 508», KeyWords 
136-and an END 504-. It should be noted that only Graphic 
Icon 144=, URL 128=and END 504=are essential and that 
only URL 128=absolutely requires an argument, that is, the 
ASCII character string or equivalent comprising the URL 
128. 

A further example of a rather extensive Visual Link 
Dataset 500 might be: 

Graphic Icon 144= Version 1.0 

KeyWords 136=web browser interface icon button visual 
internet 

bitmap compression thumbnail 

Subject 504=web„browsing 

Category =browsers 

Company-Netwave, Inc. 

USPS =440 Middlesex Road 
Tyngsboro MA 01879 

Phone=(603) 649-8633 

FAX-(603) 649-8699 

E-Mail=*gmilie fsky@netwave.com 

Description=Provider of "Visual Links Objects" for mak- 
ing cyberspace easier. 

Comment= 

PrevGraphic Icon 144= 

URL=http://www.netwavelink.com/pointreview.HTML 

SerialNo-0000000045 

SeeAlso=http://www.foobar.edu/santa.clause 

See Also=http ://www.p atriot.mil/big. guns 

Author=Gary Miliefsky 

Created=ll/10/96 

LastMod=ll/14/96 

END 504= 

NOTE: This dataset (11/14/96), is intended only for the 
purpose of debugging the various software modules 
that Netwave will soon export. IT IS NOT TO BE 
DISTRIBUTED TO OUTSIDE PARTIES EXCEPT 
AS AN EXAMPLE, AND THEN ONLY TO THOSE 
WHO HAVE SIGNED A NON-DISCLOSURE 
AGREEMENT. Although this dataset was current as of 
Nov. 17, 1996, there may be yet newer versions, with 
slightly different names. Look for all files with names 
samp*.*. 

It will be noted that this example includes text appended 
at the end, which may be helpful for developers or other 
users, and that the END 504= Designator 502 is the end of 
data of significance to the Graphic Icon 144 parser and hence 
to all computer programs that interrogate or otherwise use 
Graphic Icons 144. 
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It should also be noted that in the present implementation, 
Designators 502 are case-insensitive, meaning that any 
upper case letters and their corresponding lower case ver- 
sions are interchangeable. In addition, there may be alter- 
5 native forms for at least some Designators 502. For example, 
in order to save storage space and transmission time, at least 
some Designators 502 may be represented by four-character 
abbreviations, such as KEY= for KeyWords=. In other 
examples, the following Designators 502 may be repre- 
io sented by the corresponding following abbreviations: 

Graphic Icon 144= by Icon 144=; 

Comment- by COM-; 

Remark- by REM=; 

Date- by DAT=; 

Author= by AUT«; 

Contact- by CON-; 

Key Words- by KEY=; and, 

Subject- by SUB-. 
20 The above illustrated group or set of Designators 502 is 
intended to be a growing set, with additional Designators 
502 added as found necessary or advantageous for general 
use. It may, for example, be desirable to add a "rating 
system" designator whose arguments indicate the depiction 
25 or or reference to sex, nudity, violence or language unsuit- 
able for younger age audiences. There may also be desig- 
nators whose arguments hold information dealing with 
encryption or permissions, whereby certain parts of the 
dataset may be read or interp re table only by users possessing 
30 appropriately entitling decryption methods or passwords. 
For example, the site referenced by a URL may thus be 
visible or accessible only to privileged users. 

It is also intended that specific users may add Designators 
502 to meet their specific needs, such as failed= for the 
35 cumulative count of failures of the site to respond, found= 
for the count of successes at downloading from the site, and 
FSLS- for the count of failures since the last successful 
access to the site. 

It is intended that Visual Link Datasets 500 may be 
40 parsed, interpreted and use by relevant systems and pro- 
grams from a variety of sources. As such, it is further 
intended that Visual Link Datasets 500 conform to a com- 
monly accepted set of at least minimum rules and definitions 
so that any given Visual Link Dataset 500 may be used, at 
45 least to the minimum necessary extent to meet the intended 
purpose, by any system or program conforming to these 
rules and definitions. The following explicit rules and 
definitions, or an alternative but generally similar set of rules 
and definitions, are exemplary of such a set of rules and 
50 definitions. 

Accordingly, the following are definitions used in a pres- 
ently preferred implementation of Visual Link Datasets 500: 

A WHITESPACE is a contiguous string of non-printing 
non-ink characters, most notably space, tab, carriage return, 
55 line feed, and the implicitly understood beginning-of-file 
and end-of-file. 

A TOKEN is a contiguous string of visibly-printed ASCII 
characters that appear between successive whitespaces. 

A DESIGNATOR 502 is a token that ends with the 
60 character 

An ARGUMENT is a token not ending with appearing 
somewhere between successive designators. 

An ENTRY is logically a line of Visual Link Dataset 500 
code consisting of a Designator 502 and the following 
65 arguments), if any, up to but not including the next desig- 
nator. According to a present implementation of Visual Link 
Datasets 500, the arguments of a designator need not appear 
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on one line, that is, the whitespace separating arguments 
may be or include carriage return and/or linefeed. 

In a like manner, the rules and practices for a presently 
preferred embodiment of the present invention include that 
a Visual Link Dataset 500 is comprised of: 5 

a Graphic Icon 144= <argument(s)>; 

<cntry or entries other than the URL 128=>; 

a URL 128= plus one argument in the form of a URL- 
format web address; 1{J 

<entry or entries other than the URL 128=>; 

END 504-; and, 

<text that will be ignored by all programs, 
wherein the items delineated in < > above are regarded as 
optional. 15 

Id addition, it is presently good practice is to avoid 
arguments that end in and when this is impossible, then 
to end the argument with written in the form 
correspondingly writing the character ,A * in the form <,AA ". 

In other usages, the minimal Visual Link Dataset 500, like 20 
that depicted above, may absolutely require some designator 
and related arguments other than a URL=, such as an e-mail 
address, or phone number or postal address. 

According to the presently preferred embodiment, the 
order, number of entries, and their content in a Visual Link 25 
Dataset 500 is arbitrary except that 

(1) there must be one and only one Graphic Icon 144=, 
URL 128-and END 504-, 

(2) the very first token of a Visual Link Dataset 500 must 3Q 
be Graphic Icon 144«, except that this token is not 
required in the Visual Link Dataset 500 version 
wherein the Graphic Icon 144 and the "visual Link 
Dataset 500 are contained in a single file as the dataset 
need not point to the icon, 35 

(3) the first and probably only argument of the URL 128= 
token must be, or at least thought to be, a bona fide 
URL 128, and 

(4) the END 504= entry is the last entry or token of the file 
that will be recognized and treated by support programs 40 
that parse or use Visual Links 142. 

Also, and for purposes of human creation, examination 
and/or editing, it is preferable that a Visual Link Dataset 500 
be comprised of human readable characters, such as ASCII 
printing characters, blank, carriage return and line feed. 45 

As has been described herein, a Visual Link Dataset 500 
may be generated automatically by a capture engine, such as 
a Visual Link Capture Engine 138, whereby Web sites are 
interrogated and the Graphic Icon 144 bitmap picture 
selected and other data gleaned by downloading and scan- 50 
ning the HTML file from each site. Since a Visual Link 
Dataset 500 is encoded as an ASCII file, however, a Visual 
Link Dataset 500 may alternatively be generated manually, 
for example, by a person using a text editor such as a normal 
word -processing program. 55 

Having considered Graphic Icons 144 and Visual Link 
Datasets 500 herein above, the following will consider the 
means for association of a Graphic Icon 144 and a Visual 
Link Dataset 500 to construct a Visual Link 142. Association 
of Picture and Graphic Icon 144 Dataset. 60 

As discussed, a Visual Link 142 consists essentially of a 
picture, that is, a Graphic Icon 144, and a Visual Link 
Dataset 500, which must be associated with one another to 
form an entity referred to herein as a Visual Link 142. and 
a Graphic Icon 144 dataset This may be accomplished in 65 
several ways, as illustrated in FIGS. 5D through 5G., the 
presently preferred method being illustrated in FIG. FIG. 5F. 
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Referring first to FIG. 5D, in this embodiment the Graphic 
Icon 144 picture is described in a typical graphic or image 
file format, such as JPEG or GIF, with pixel data compressed 
or not, and is this file is completely isolated from the Visual 
Link Dataset 500, forming a separate file, and is unaltered. 
The association between the Graphic Icon 144 file and the 
Visual Link Dataset 500 file is maintained on a higher level, 
for example, in a database or link file structure, identified in 
FIG. 5D as Link 510 that contains at least the relative or 
absolute Pathnames 512A and 512B of the two files, or the 
equivalent. 

Advantages of this embodiment is that the picture file may 
be used in any and all of the possible ways that are relevant 
to the format of the encoded bitmap picture, as can be the 
Visual Link Dataset 500 file, and the Graphic Icon 144 
picture file and the Visual link Dataset 500 file are indi- 
vidually created or modified more easily. A disadvantage of 
this embodiment, however, is that three items, the Graphic 
Icon 144 file, the Visual Link Dataset 500 file and the 
database or file structure relating the two, must be main- 
tained in agreement, and that more care and/or utilities are 
thereby needed to maintain between the components of the 
Visual Link 142. 

An alternate embodiment is illustrated in FIG. 5E wherein 
it is shown that the Graphic Icon 144 picture file and the 
Visual Link Dataset 500 file are again separate files but that 
the association between them is indicated in a Designator 
502 in the Visual Link Dataset 500, such as 'PIC 514='. In 
this embodiment, the argument of PIC 514= is the absolute 
or relative pathname, or equivalent for the Graphic Icon 144 
picture file, thereby eliminating the database of file structure 
used in the embodiment of FIG. 5D to relate the Graphic 
Icon 144 and the Visual Link Dataset 500. The advantages 
of the embodiment illustrated in FIG. 5E are the same as for 
the embodiment illustrated in FIG. 5D, and the disadvan- 
tages are similar in that there are still two files whose 
locations and/or contents must be maintained consistently, 
although the disadvantages are reduced with respect to the 
embodiment of FIG. 5D. 

Yet another embodiment, and ia this instance the pres- 
ently preferred embodiment, is illustrated in FIG. 5F. As 
illustrated therein, the Visual Link Dataset 500 is appended 
to an otherwise unaltered Graphic Icon 144 picture to form 
a single File 516, which may typically be an image file, 
wherein a Length-of-Dataset Number 518A is appended to 
the Visual Link Dataset 500 in the single File 516 and 
represents the location of the beginning of the Visual Link 
Dataset 500 in the File 516 relative to the end of the File 516, 
and thereby the end of the Graphic Icon 144 picture in the 
File 516 relative to the end of the File 516. The beginning 
of the Visual Link Dataset 500 is thereby found by seeking 
to end-of-file minus size of Length-of-Dataset Number 
518A, interpreting the Length-of-Dataset Number 518A 
found at that location, and seeking backward this number of 
bytes to find the beginning of the Visual Link Dataset 500 
component, and the end of the Graphic Icon 144 picture 
component. 

In the present embodiment, Length-of-Dataset Number 
518A is stored as a fixed number of ASCII characters, such 
as four, and high digit first, rather than as an n-byte number, 
thereby avoiding possible confusion that could arise from 
differences in byte order of words and double words, and 
differences in bit order in a byte, in different systems. In yet 
other implementations of the embodiment shown in FIG. 5 F, 
Length-of-Dataset Number 518A may be accompanied by a 
URL Pointer 518B identifying the location of the URL 128 
Designator 502 in the file and an END Pointer 518C 
identifying the location of the END 504- Designator 502 in 
the file. 
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It will be apparent that the advantages of this embodiment 
are that there is only one file to maintain, move or copy and 
that the entire file can be used for most purposes relevant to 
the display of a non-augmented picture file, such as a display 
on a screen. It should be noted, however, that if Graphic 5 
Icons 144 are dealt with only by programs designed spe- 
cifically for this purpose, this problem should not arise. 

Possible disadvantages could arise in this embodiment 
due to the structure of the file wherein the actual length of 
the composite file is different from the length of the file as 10 
may be declared within the image file subpart as the user or 
program constructing this part has generally not anticipated 
an appendment. That is, the declared length of the composite 
file would typically be based upon the size of one component 
of the file, such as the Graphic Icon 144 component of the 15 
file if the composite file is treated as a normal picture file. If 
it is operated upon by a utility that operates upon the file 
based upon the declared and expected end-of-file length, the 
utility may read and operate upon the expected rather than 
actual length of the file. As a consequence, portions of the 20 
file, such as the Visual Link Dataset 500 component, may 
not be operated upon and may, for example, be truncated in 
a copy or move operation. The same problem may arise 
when a person or program copies the file into a buffer that 
accommodates only the self-declared size of the picture file 25 
as declared in the image part of the file. 

In a related potential problem, any editor that is selected 
for use in modifying the file must be chosen so as to not 
make end of line or file modifications or other conversions 
that might alter the data stored in, for example, the Graphic 30 
Icon 144 picture component. In this instance, it may be 
preferable that the Visual Link Dataset 500 component of the 
composite file be split off from the composite for editing, 
and the two files subsequently re-concatenated. 

It will be noted, however, that a system's file system will 35 
generally know or recognize the actual complete size or 
length of the file and it is only the length indicator or 
indicators within the image part of the file that may give 
misleading indications of the length of the composite file. 
Programs or utilities that do not rely only on the self 40 
declared length of the image part of the file will therefore 
generally avoid such problems. 

In a fourth alternate embodiment, many bitmap picture 
file protocols, such as those for JPEG and GIF, accommo- 
date application specific extensions or inclusions and, in 45 
these instance, the Visual Link Dataset 500 could be added 
to the Graphic Icon 144 image file as an application specific 
extension. This embodiment is advantageous in correspond- 
ing to established standards of the field of art and in relaxing 
or avoiding at least some of the caveats discussed above in 50 
appending a Visual Link Dataset 500 component to an image 
file. A disadvantage, however, is that utility programs for 
dealing with the Visual Link Dataset 500 components of the 
resulting composite files must be custom-designed or 
adjusted to each such image file format. 55 

Finally, those of skill in the relevant arts will appreciate 
that there are still further methods for associating the two 
files. For example, a common basename with different 
extensions, such as 'el ephant.gif and 'elephant, txt', would 
have essentially the advantages and disadvantages discussed 60 
above with regard to the embodiment illustrated in FIG. 5E. 

7. Hash Protection of Files Such as Visual Links 
142 (FIGS. 6A and 6B) 

Referring to FIG. 6A, therein is represented a Hashing 65 
Protection Mechanism 600 implemented, for example, in a 
Local System 112 or a Server System 114 for encoding 
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computer system files, such as a Visual Link 142 file, to 
allow detection of the unauthorized construction or modifi- 
cation of such files. 

Hashing Protection Mechanism 600 performs this finction 
for a File 602 by reading a Encrypting Part 604 of the Data 
606 of the File 602, wherein Encrypting Part 604 comprises 
at least a part of Data 606, and generating a "Hash" 
Encrypted Value 608 by performing a known bashing opera- 
tion on Encrypting Part 604. Hashing Protection Mechanism 
600 then associates the Hash Encrypted Value 608 with the 
File 602, for example, by embedding Hash Encrypted Value 
608 as a Stored Hash Encrypted Value 608S in a Storing Part 
610 of Data 606 that is not included or contained within 
Encrypting Part 604. For these purposes, and as illustrated in 
FIG. 6A, Hashing Protection Mechanism 600 includes a File 
Reader 612 for identifying and reading an Encrypting Part 
604 of a File 602, a Hashing Mechanism 614 for performing 
the hashing operation on the Encrypting Part 604, and a File 
Writer 616 for writing Stored Hash Encrypted Value 608S 
into Storing Part 610. 

Thereafter, Hashing Protection Mechanism 600 may 
determine whether there has been an unauthorized construc- 
tion or modification of a File 602 has been altered by reading 
Encrypting Part 604, performing the hashing operation on 
Encrypting Part 604 to determine a Confirmation Hash 
Encrypted Value 608 C, and comparing the Confirmation 
Hash Encrypted Value 608C with the Stored Hash Encrypted 
Value 608S from Storing Part 610. For these purposes, and 
as illustrated in FIG. 6A, Hashing Protection Mechanism 
600 again includes File Reader 612 for reading Encrypting 
Part 604 and Hashing Mechanism 614 for generating a 
Confirmation Hash Encrypted Value 608C from Entrypting 
Part 604. Hashing Protection Mechanism 600 further 
includes Comparator 618 for reading the Stored Hash 
Encrypted Value 6908S from Storing Part 610 comparing 
the Stored Hash Encrypted Value 608S and the Confirmation 
Hash Enctrypted Value 608C to determine whether the File 
602 has be properly constructed or modified by an autho- 
rized user or system. 

It will be apparent that, because the specific hashing 
algorithm used in the hashing operation is known or avail- 
able only to users or systems authorized to construct or 
modify Files 602, only an authorized user or system will be 
able to generate a Confirmation Hash Encrypted Value 608C 
from Encrypting Part 604 that is equal to the Stored Hash 
Encrypted Value 608S. If the values are equal, then the File 
602 has not been constructed or modified without proper 
authorization and, if the values are not equal then the File 
602 has been constructed or modified with a hashing algo- 
rithm different from that available and known to the autho- 
rized users or systems. This method assumes, of course, that 
the hashing algorithm is held in secrecy, which is a common 
requirement for almost all protection mechanisms, and that 
the algorithm is sufficiently complex to avoid accidental or 
easy discovery, which is a general requirement of all pro- 
tection mechanisms and a common characteristic of many 
hashing algorithms. 

According to the present invention, Storing Part 610 in 
which Stored Hash Encrypted Value 608C is stored is 
comprised of portions of Data 606 that are not only separate 
from Encrypting Part 604 but also are characterized by 
having alternate forms of expression, that is, different states, 
that are semantically equivalent to a user or a computer 
system, and wherein the alternate, semantically equivalent 
states of the Data 606 in Storing Part 610 are used to encode 
the Stored Hash Encrypted Value 608C. 

For example, there may be portions of Data 606 com- 
prised of legible words or pseudo-words or abbreviations in 



WEST 



US 6,181,838 Bl 



61 



62 



10 



25 



English text that, for their primary purpose are semantically 
equivalent to a human user, such as "END" and "end", or 
that are treated in a case insensitive manner by a system. 
Assuming, for example, that a Stored Hash Encrypted Value 
608C is comprised of or expressed as a binary string, such 
as "101", and that an upper case letter is used to indicate a 
"1" while a lower case letter is used to represent a "0", this 
Stored Hash Encrypted Value 608C could be encoded in the 
word "end" as "EnD" so long as this expression of "end" 
was, as necessary for its primary purpose, semantically 
equivalent to a user or that a system treated the word in a 
case insensitive manner. Thus, "end", "End", "eNd", "enD", 
"ENd", EnD", eND", and "END" could all be used to 
encode different Stored Hash Encrypted Values 608C if 
these varients of "end" had equivalent meanings to a user or 
a system, as necessary for the primary purpose of the word. ]5 

Therefore, minute differences in ASCII code, for 
example, may signify bits of a Stored Hash Encrypted Value 
608C, with one bit of the Stored Hash Encrypted Value 608C 
being represented by each ASCII character or string of 
identically treated ASCII characters, so long as the use of 20 
alternate versions of the characters or strings of characters 
did not interfere with the primary functions of the characters. 
Also, and as described above, these characters or strings of 
characters may not or should not be used as the Encrypting 
Part 604. In addition, the parts of a file that are used for 
Encrypting Part 604 and for Storing Part 610 should be 
easily identifiable and locatable by a system. 

This method of the present invention for encoding a hash 
encrypted value generated from portions of the data of a file 
and stored in other portions of the data of the file having 
alternate but semantically equivalent expressions that may 
be used to encode the hash encrypted valued is illustrated in 
FIGS. 6B-1 and 6B-2 wherein FIG. 6B-1 represents an 
original file and FIG. 6B-2 represents the same file with the 
bits of the hash encrypted value, which are listed in a column 
at the left of the file, encoded in the corresponding case- 
distinguishing bits of the ASCII characters used to encode 
the hash encrypted value. It will be noted that the substrings 
of characters that are considered as valid for encoding by 
alternate expression are all contiguous substrings of letters 
terminated by the "=" sign. It will also be noted that, for 
cosmetic reasons, in each such substring all characters but 
the first share the same change, that is, the same alternate 
form of expression and collectively represent one bit of the 
hash encrypted value, and that the beginning character of 
each substring is left in the upper case state. 

It may be apparent, therefore, that the hashing protection 
method and mechanism of the present invention is no more 
complex and requires no more effort, for example, in main- 
taining the secrecy of the hashing algorithm, than alternate 
protection mechanisms. It is further apparent that the hash- 
ing protection method and mechanism of the present inven- 
tion is advantageous in that the encrypted hash value is 
stored in a non-intrusive manner that takes no extra space in 
a file and does not change the encrypted hash value of the file 
and in a manner that does not impede human or system 
interpretation of the data in the file. 

Finally, while the invention has been particularly shown 
and described with reference to preferred embodiments of 
the apparatus and methods thereof, it will be also understood 
by those of ordinary skill in the art that various changes, 
variations and modifications in form, details and implemen- 
tation may be made therein without departing from the spirit 
and scope of the invention as defined by the appended 
claims. Therefore, it is the object of the appended claims to 
cover all such variation and modifications of the invention as 
come within the true spirit and scope of the invention. 
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What is claimed is: 

1. For use in a data processing system including a memory 
for storing programs and files containing data and a proces- 
sor for operating on the data under control of the programs, 
a capture engine for extracting graphics information from a 
data file and generating a corresponding graphic icon form- 
ing a displayable image representing the graphics 
information, comprising: 

a grayscale image generator for receiving an original 
image and generating a corresponding grayscale image 
containing brightness values representing the original 
image, 

an edge image mechanism for receiving the grayscale 
image and generating a corresponding edge image 
representing areas of visually significant graphic struc- 
ture in the grayscale image as represented by areas of 
change in the brightness values, 

a candidate region search mechanism for receiving the 
edge image and identifying initial candidate regions of 
the edge image representing visually significant areas 
of the original image, and 

a candidate region adjustment and comparator mechanism 
for selecting a candidate region to be used in generating 
a corresponding graphic icon and for adjusting the 
selected candidate region to conform to predetermined 
dimensions for a graphic icon. 

2. The capture engine of claim 1, wherein the grayscale 
image generator comprises: 

a grayscale lookup table for storing grayscale brightness 
values corresponding to each of the possible brightness 
values of image elements in original images, 
a grayscale converter for 

reading the brightness values of the image elements of 

the original image, 
reading the corresponding grayscale brightness values 

stored in the grayscale lookup table, and 
generating a corresponding grayscale image wherein 
each image element of the original image is repre- 
sented by a grayscale image element having the 
grayscale brightness value corresponding to the 
brightness value of the image element. 

3. The capture engine of claim 2, wherein each image 
element of the original image is represented by a set of color 
values, the set of color values of an image element of the 
original element representing the color and brightness value 
of the image element, and wherein: 

the grayscale lookup table contains a grayscale brightness 
value corasponding to each of the possible sets of 
color values of image elements in original images, and 
the grayscale converter is responsive to the set of color 
values of each image element of the original image for 
reading the corresponding grayscale brightness values 

stored in the grayscale lookup table, and 
generating a corresponding grayscale image wherein 
each image element of the original image is repre- 
sented by a grayscale image element having the 
grayscale brightness value corresponding to the set 
of color values of the image element. 

4. The capture engine of claim 2, wherein: 

the grayscale image generator generates each image ele- 
ment of the grayscale image from a plurality of corre- 
sponding elements of the original image so that the 
number of image elements in the grayscale image is 
proportionally reduced from the number of image ele- 
ments in the corresponding original image. 

5. The capture engine of claim 1, wherein the edge image 
mechanism comprises: 
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an edge image generator for receiving an input grayscale 
image and generating a corresponding output edge 
image wherein each image element of an output edge 
image is represented by an edge value proportionate to 
the difference between the brightness value of the 5 
corresponding image element in the input grayscale 
image and the brightness values of the neighboring 
image elements of the corresponding image element in 
the input grayscale image, wherein 
an output edge image thereby represents areas of visu- io 
ally significant graphic structure in the input gray- 
scale image as represented by areas of change in the 
brightness values of the input grayscale image. 

6. The capture engine of claim 5, wherein the edge image 
mechanism further comprises: 15 

a lowpass filter connected between the grayscale image 
generator and the edge image generator for receiving a 
grayscale image and generating a corresponding fil- 
tered grayscale image of reduced resolution and pro- 
vides the filtered grayscale image to the edge image 20 
generator as the correspond input grayscale image to 
the edge image generator, wherein 
each image element of a filtered grayscale image is 
determined as a proportioned summation of the 
brightness values of selected neighboring image ele- 25 
ments of the image element of the grayscale image, 
so that a filtered grayscale image thereby represents 
visual structures of significant extents of the gray- 
scale image. 

7. The capture engine of claim 5, wherein: 30 
the edge image generator generates each image element of 

an output edge image from a plurality of corresponding 
elements of the input grayscale image so that the 
number of image elements in the output edge image is 
proportionally reduced from the number of image ele- 35 
ments in the corresponding grayscale input image. 

8. The capture engine of claim 1, wherein the candidate 
region search mechanism comprises: 

a rectangular sum array generator for receiving an output 4Q 
edge image and generating a corresponding rectangular 
sum array having a sum element corresponding to each 
image element of the output edge image wherein each 
sum element represents the sum of the edge values of 
the image elements of a region of the output edge image 45 
bounded by a coordinate location of the output edge 
image and the output edge image element correspond- 
ing to the sum element, and 

a rectangular sum search mechanism for examining the 
summed edge values represented in each of the sum 50 
elements of the rectangular sum array and identifying at 
least one initial candidate region having a high summed 
edge value, each initial candidate region thereby rep- 
resenting and corresponding to a region of the original 
image having a significant visual structure. 55 

9. The capture engine of claim 1, wherein the candidate 
region adjustment and comparator mechanism comprises: 

an edge adjuster for receiving the at least one initial 
candidate region and 

generating from each initial candidate region a corre- 60 
spending grown candidate region by adjusting the 
extents of each initial candidate region to include all 
portions of significant visual structures having a part 
thereof contained within the initial candidate region, 

generating from each grown candidate region a corre- 65 
sponding edge adjusted candidate region by adjust- 
ing the extents of each grown candidate region so 



that each edge adjusted candidate region conforms to 
one of a set of predetermined extents for graphic 
icons, 

comparing each edge adjusted candidate region with 
others of the edge adjusted candidate regions and 
selecting the edge adjusted candidate region having 
the highest amount of significant visual structure, 
comparing each selected edge adjusted candidate 
region with previously existing graphic icons and, 
when the selected edge adjusted candidate region is 
similar to a previously existing graphic icon, 
selecting a next one of the at least one edge 
adjusted candidate regions, and, 
when a selected candidate region is distinguished 
from the previously existing graphic icons, iden- 
tifying the corresponding region of the original 
image for use as a graphic icon. 
10. For use in a data processing system including a 
memory for storing programs and files containing data and 
a processor for operating on the data under control of the 
programs, a capture engine for extracting graphics informa- 
tion from a data file and generating a corresponding graphic 
icon forming a displayable image representing the graphics 
information, comprising: 

a grayscale image generator for receiving an original 
image and generating a corresponding grayscale image 
containing brightness values representing the original 
image, including 

a grayscale lookup table for storing grayscale bright- 
ness values corresponding to each of the possible 
brightness values of image elements in original 
images, 
a grayscale converter for 

reading the brightness values of the image elements 

of the original image, 
reading the corresponding grayscale brightness val- 
ues stored in the grayscale lookup table, and 
generating a corresponding grayscale image wherein 
each image element of the original image is rep- 
resented by a grayscale image element having the 
grayscale brightness value corresponding to the 
brightness value of the image element, 
an edge image mechanism for receiving the grayscale 
image and generating a corresponding edge image 
representing areas of visually significant graphic struc- 
ture in the grayscale image as represented by areas of 
change in the brightness values, including 
an edge image generator for receiving an input gray- 
scale image and generating a corresponding output 
edge image wherein each image element of an output 
edge image is represented by an edge value propor- 
tionate to the difference between the brightness value 
of the corresponding image element in the input 
grayscale image and the brightness values of the 
neighboring image elements of the corresponding 
image element in the input grayscale image, wherein 
an output edge image thereby represents areas of 
visually significant graphic structure in the input 
grayscale image as represented by areas of change 
in the brightness values of the input grayscale 
image, and 

a candidate region search mechanism for receiving the 
edge image and identifying initial candidate regions of 
the edge image representing visually significant areas 
of the original image, including 
a rectangular sum array generator for receiving an 
output edge image and generating a corresponding 
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rectangular sum array having a sum element corre- 
sponding to each image element of the output edge 
image wherein each sum element represents the sum 
of the edge values of the image elements of a region 
of the output edge image bounded by a coordinate 
location of the output edge image and the output 
edge image element corresponding to the sum 
element, and 

a rectangular sum search mechanism for examining the 
summed edge values represented in each of the sum 
elements of the rectangular sum array and identify- 
ing at least one initial candidate region having a high 
summed edge value, each initial candidate region 
thereby representing and corresponding to a region 
of the original image having a significant visual 
structure, and 

a candidate region adjustment and comparator mechanism 
for selecting a candidate region to be used in generating 
a corresponding graphic icon and for adjusting the 
selected candidate region to conform to predetermined 
dimensions for a graphic icon. 

11. The capture engine of claim 10, wherein each image 
element of the original image is represented by a set of color 
values, the set of color values of an image element of the 
original element representing the color and brightness value 
of the image element, and wherein: 

the grayscale lookup table contains a grayscale brightness 
value corresponding to each of the possible sets of 
color values of image elements in original images, and 
the grayscale converter is responsive to the set of color 
values of each image element of the original image for 
reading the corresponding grayscale brightness values 

stored in the grayscale lookup table, and 
generating a corresponding grayscale image wherein 
each image element of the original image is repre- 
sented by a grayscale image element having the 
grayscale brightness value corresponding to the set 
of color values of the image element. 

12. The capture engine of claim 10, wherein: 

the grayscale image generator generates each image ele- 
ment of the grayscale image from a plurality of corre- 
sponding elements of the original image so that the 
number of image elements in the grayscale image is 
proportionally reduced from the number of image ele- 
ments in the corresponding original image. 

13. The capture engine of claim 10, wherein the edge 
image mechanism further comprises: 

a lowpass filter connected between the grayscale image 
generator and the edge image generator for receiving a 



grayscale image and generating a corresponding fil- 
tered grayscale image of reduced resolution and pro- 
vides the filtered grayscale image to the edge image 
generator as the correspond input grayscale image to 
5 the edge image generator, wherein 

each image element of a filtered grayscale image is 
determined as a proportioned summation of the 
brightness values of selected neighboring image ele- 
ments of the image element of the grayscale image, 
10 so that a filtered grayscale image thereby represents 

visual structures of significant extents of the gray- 
scale image. 
14. The capture engine of claim 12, wherein: 
the edge image generator generates each image element of 
15 an output edge image from a plurality of corresponding 
elements of the input grayscale image so that the 
number of image elements in the output edge image is 
proportionally reduced from the number of image ele- 
ments in the corresponding grayscale input image. 
20 15. The capture engine of claim 10, wherein the candidate 
region adjustment and comparator mechanism comprises: 
an edge adjuster for receiving the at least one initial 
candidate region and 

generating from each initial candidate region a corre- 
25 sponding grown candidate region by adjusting the 

extents of each initial candidate region to include all 
portions of significant visual structures having a part 
thereof contained within the initial candidate region, 
generating from each grown candidate region a corre- 
30 sponding edge adjusted candidate region by adjust- 

ing the extents of each grown candidate region so 
that each edge adjusted candidate region conforms to 
one of a set of predetermined extents for graphic 
icons, 

35 comparing each edge adjusted candidate region with 
others of the edge adjusted candidate regions and 
selecting the edge adjusted candidate region having 
the highest amount of significant visual structure, 
comparing each selected edge adjusted candidate 
40 region with previously existing graphic icons and, 

when the selected edge adjusted candidate region is 
similar to a previously existing graphic icon, 
selecting a next one of the at least one edge 
adjusted candidate regions, and, 
45 when a selected candidate region is distinguished 

from the previously existing graphic icons, iden- 
tifying the corresponding region of the original 
image for use as a graphic icon. 
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ABSTRACT 



A digital image management system is described that 
includes a content analyzer that analyzes an image to extract 
content data from the image. The content data of an image 
include face feature data. The digital image management 
system also includes an image database that is coupled to the 
content analyzer to store pixel data of each of a number of 
images and the content data of each of the images. A search 
engine is also provided in the digital image management 
system. The search engine is coupled to the image database 
and the content analyzer to compare the content data of the 
images with that of an input image such that any image 
similar to the input image can be identified from the image 
database without retrieving the pixel data of the image from 
the image database. A method of extracting feature data of 
a face in an image is also described. 

7 Claims, 6 Drawing Sheets 
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DIGITAL IMAGE MANAGEMENT SYSTEM 
IN A DISTRIBUTED DATA ACCESS 
NETWORK SYSTEM 

BACKGROUND OF THE INVENTION 5 

1. Field of the Invention 

The present invention pertains to digital image manage- 
ment. More particularly, this invention relates to a digital 
image management system that automatically extracts con- 10 
tent information from digital images and associates the 
content information with the digital images such that the 
digital images can be effectively and efficiently searched and 
retrieved. 

2. Description of the Related Art 15 
As is known, photographic prints have traditionally been 

used for visual presentation. The photographic prints are 
desirable because photographic papers are relatively inex- 
pensive and easily portable. However, when a viewer needs 
to locate a desired photographic picture among a large 20 
number of pictures, the viewer has to individually view each 
of the pictures until the desired picture is identified. This is 
typically a very time consuming process. The process can 
also be very frustrating especially when the desired picture 
needs to be identified quickly. 25 

In order to reduce the time one needs to find the desired 
picture among a large number of pictures, the pictures can be 
organized into a number of different categories. The orga- 
nization typically involves labeling containers that store the 
pictures and building index lookup tables so that the viewer 30 
can locate the appropriate container or containers for the 
desired category relatively quickly. The disadvantage of this 
prior art approach is that a lot of time is typically spent on 
categorizing the pictures and arranging them according their 
categories. Moreover, even with this approach, the task of 35 
searching for the desired picture is still time consuming and 
difficult The viewer needs to remember which picture 
belongs to which category in order to find the desired 
picture. The viewer also needs to go through the pictures 
within the desired category in order to identify the desired 40 
picture. Furthermore, each picture is assigned to one cat- 
egory. 

In addition, this prior art image management system may 
not function properly if pictures are misplaced or mis- 45 
categorized. When this occurs, the desired picture may not 
be identified unless all or most of the pictures have been 
viewed. In this case, more time is needed to find the desired 
picture. 

With the advance of computer technologies, it is now 50 
possible to store an image in a computer system. This can be 
done using a scanner that converts the image on a print or 
film into a digital image. The digital image includes image 
pixels (or dots) arranged in a matrix. The data for each image 
pixel are then stored in the computer. The digital image can 55 
be retrieved for display from the computer. The digital 
image can also be digitally altered in the computer. 
Moreover, the digital image can be transmitted to a remote 
computer for viewing via a communication network. This 
allows real time viewing of the image. 60 

However, searches of these digital images based on their 
image contents in the computer are still nearly impossible. 
In order to identify a desired image stored in the computer, 
the computer can only compare the image pixel data of an 
input image with the corresponding pixel data of each of the 65 
images stored in the computer. The comparison typically 
takes a very long time to complete and uses a lot of system 



2 

resources (e.g., memory and communication bandwidth). 
This is because the computer has to compare each image 
pixel data of the input image with the corresponding image 
pixel data of each of the stored images. A match occurs when 
each pixel data of the input image is similar the correspond- 
ing pixel data of a stored image. Using this method, unless 
the two compared images are very similar, it is very difficult 
to find a match between any two images. This is especially 
true because noise and other positional errors may be 
introduced into the image data during scanning. 

SUMMARY OF THE INVENTION 

One feature of the present invention is to allow image 
search and retrieval in a data processing system. 

Another feature of the present invention is to provide a 
digital image management system that automatically 
extracts and stores content data of digital images such that 
effective and efficient search and retrieval of the digital 
images can be achieved. 

A further feature of the present invention is to allow 
efficient image search and retrieval in a distributed data 
access network system. 

A digital image management system is described that 
includes a content analyzer that analyzes an image to extract 
content data firom the image. The content data include face 
feature data. The digital image management system also 
includes an image database that is coupled to the content 
analyzer to store pixel data of each of a number of images 
and the content data of each of the images. A search engine 
is also provided in the digital image management system. 
The search engine is coupled to the image database and the 
content analyzer to compare the content data of the images 
stored in the image database with the content data of an input 
image such that any image similar to the input image can be 
identified from the image database without retrieving the 
pixel data of the image from the image database. 

A method of extracting feature data of a face in an image 
is also described. The method first detects if the image 
contains the face using a neural network-based face detec- 
tion system. Then the feature data of the face are computed 
using a principle component analysis system. 

In addition, the digital image management system can be 
implemented in a distributed data access network system 
that includes a number of data service systems. In this case, 
the content analyzer is located within one of the data service 
systems and the image database includes a number of 
databases, each being located within one of the data service 
systems. The search engine may include a number of search 
and retrieval engines, each being located within one of the 
data service systems. 

Other features and advantages of the present invention 
will become apparent from the following detailed 
description, taken in conjunction with the accompanying 
drawings, illustrating by way of example the principles of 
the invention. 

BRIEF DESCRIPTION OF THE DRAWINGS 

FIG. 1 shows a distributed data access network system 
that has a number of data service systems, each of which 
having an image database that is part of a digital image 
management system in accordance with one embodiment of 
the present invention. 

FIG. 2 shows the data format of the digital images and 
their content data stored in each of the image databases. 

FIG. 3 is a block diagram of the digital image managing 
system in accordance with one embodiment of the present 
invention. 
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FIG. 4 shows the content analyzer of the digital image 100, in association with the pixel data of the respective 

management system of FIG. 3, wherein the content analyzer image. The content data and the pixel data of a particular 

includes a face detection system and a face feature extrac- image may be stored in different image databases. When this 

tion system. occurs, the logical links (e.g., hyper-links) between the 

FIG. 5 shows the flow chart diagram of the process of 5 content data and the pixel data are stored into one of the 

detecting faces in an image and extracting face feature data image databases. The content analyzer 102 performs this 

from each of the faces detected by the face detection system function using a linking system 204 (see FIG. 4). 

and the face feature extraction system of FIG. 4. When the image management system 100 receives an 

FIG. 6 shows the flow chart diagram of the process of image for browsing, querying, or comparison, the content 

searching and retrieving images from the image database by 10 data of the new image are extracted and compared with that 

the search and retrieval engine of FIG. 3. of the images stored in the image databases of the image 

management system 100 based on a predetermined similar- 

DETAILED DESCRIPTION OF THE ity measure such that any image similar to the input image 

INVENTION cm De identified from the image databases without viewing 

FIG. 1 shows a data access network system 50 within 15 tDe images. The search results can be ranked, for example, 

which a digital image management system 100 (shown in according to similarity scores. 

FIG. 3 in block diagram form) in accordance with one In addition, the image management system 100 can 

embodiment of the present invention is implemented. The receive the content data (instead of the actual image) to 

image management system 100 automatically extracts con- search for any image stored in the image databases of the 

tent data from digital images to allow the digital images 20 system 100 that has similar content data. The comparison is 

stored in image databases of the data access network system done by a search engine 104 of the image management 

50 to be effectively and efficiently searched and retrieved. system 100. In one embodiment, the search engine 100 

The content data include face feature data and other kinds of retrieves the pixel data of the similar images after the search 

image content data. When the image management system is completed. In another embodiment, the search engine 104 

100 is implemented in a distributed network system, some of 25 only retrieves the content data of the similar images to 

the components or modules of the image management produce the ranked list. In this case, the search engine 104 

system 100 include functionally identical units, each being retrieves the pixel data of the similar images only when 

within an individual system of the distributed network display of the images is requested by the user, 

system. ^ Moreover, the image management system 100 also 

Alternatively, the image management system 100 can be includes a labeling capability that allows the user to asso- 

implemented within other kind of systems. For example, the ciate personal information (e.g., name) with a face in an 

image management system 100 can simply be implemented image. This is done by an auto -annotation module 107 of the 

in a single computer system. Furthermore, the image man- image management system 100. The image management 

agement system 100 can be implemented in a computeren- 35 system 100 will be described in more detail below, also in 

abled DVD (Digital Versatile Disc) browser/player, a conjunction with FIGS. 1-6. 

computer-enabled camera or scanner, or other kinds of Referring now to FIG. 1, the data access network system 

appliances. 50 that embodies the digital image management system 100 

As will be described in more detail below and in accor- is described in more detail. In one embodiment, the data 

dance with one embodiment of the present invention, the 40 access network system 50 is a distributed or federated 

image management system 100 allows content-based image network system that spans multiple independently adminis- 

search, querying, and browsing. The image management trated control domains. This means that the system 50 is an 

system 100 allows automatic extraction of content data of open system that includes different administrative entities 

digital images. This allows the digital images to be effi- cooperating together as a network. Alternatively, the data 

ciently and effectively searched in accordance with their 45 access network system 50 can have other network architec- 

content information. This also minimizes the effort required tures. For example, the data access network system 50 may 

to catalog the digital images with respect to their contents. be a closed network system. 

In other words, the image management system 100 effec- The data access network system 50 includes a number of 
tively provides a user with access to a very large set of data service systems 10 and 20 through 20n that are con- 
content-searchable images. 50 nected together via an interconnect network 21. Each of the 

The image management system 100 analyzes an image to data service systems 10 and 20-20/1 can be an Internet/ 
extract the content data from the image with a content Intranet service system in one embodiment. This means that 
analyzer 102. The content data of the image include color each of the data service systems 10 and 20-20/a contains the 
data of the image, texture data of the image, face feature data same logical structure. Therefore, only the data service 
of the image, shape and position data of objects within the 55 system 10 will be described in more detail below, 
image, and other feature data of the image. In addition, the The data service system 10 is further connected to sub- 
content data may also include text description data of the scriber sites 11-13 via an interconnect network 34. Included 
image. The content data can be referred to as meta-data. The in the data service system 10 are a router 37, content servers 
content analyzer 102 extracts the face feature data by first 35, proxy servers 31, DNSs (Domain Name Servers) 32, and 
detecting if the image contains one or more faces using a eo DHCP (Dynamic Host Configuration Protocol) servers 33. 
face detection system 201 (see FIG. 4). Then the content The router 37 is used for routing data to and from the 
analyzer 102 computes the feature data of each of the faces subscriber sites 11-13 upon receiving a request from a 
in the image using a face feature extraction system 202 (see subscriber/user. The router 37 also directs data traffic to the 
FIG. 4). appropriate servers 31-33 and 35, or to the other data service 

The content data of an image are then stored in one of the 65 systems 20-20w. The content servers 35 support a variety of 

image databases (e.g., the image databases 36 and 40 Internet applications, including World Wide Web access, 

through 40/i of FIG. 1) of the image management system electronic mail, bulletin boards, news groups and FTP 
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access. The proxy servers 31 may be used to enhance 
security of accesses to and from the subscriber sites U-13, 
as well as to speed up Internet access to other data service 
systems 20-20« from the data service system 10 by caching 
frequently accessed data locally. AH of the servers 31-33 
and 35 are well known in the art. 

The data service system 10 can be implemented by a 
single computer system or a number of computer systems 
connected together. When the data service system 10 is 
implemented by a single computer system, each of the 
components 31-33 and 35-37. is within the computer sys- 
tem. When the data service system 10 is implemented by a 
number of computer systems, each computer system can 
include one or more of the components 31-33 and 35-37. 

The data service system 10 provides, for example, Inter- 
net or Intranet service to subscriber sites 11 through 13 via 
the interconnect network 34. This means the data access 
network system 50 is an Internet or Intranet network system. 
FIG. 1 only shows the subscriber sites 11-13 for illustration 
purposes. In practice, many more sites may be connected to 
the data service system 10. 

The subscriber sites 11-13 may include terminals of 
subscribers/users located at the residences, schools, or 
offices of the subscribersMsers. The interconnect network 34 
can be any known network. For example, the interconnect 
network 34 can be the telephone network system. 

Each of the subscriber sites 11-13 may include a personal 
computer, a network computer, a notebook computer, a 
workstation, a mainframe computer, a supercomputer, or any 
other type of data processing system. Moreover, each of the 
subscriber sites 11-13 may be a multi-media web TV or a 
web-enabled smart video player or scanner. 

As can be seen from FIG. 1, each of the data service 
systems 10 and 20-20/? includes an image database. For 
example, the data service system 10 includes the image 
database 36 and the data service system 20 includes the 
image database 40. The image databases 36 and 40-40/1 are 
part of the image management system 100 of FIG. 3 when 
the image management system 100 is implemented within 
the data access network system 50. 

Each of the image databases 36 and 40-40/j can be 
accessed from, for example, one of the subscriber sites 
11-13 via the content servers of the respective data service 
system. Each of the image databases 36 and 40-40n stores 
a number of digital images and their content data. Each of 
the digital images stored is, for example, in the pixel matrix 
format in the respective image database. The content data of 
each image are associated with the respective digital image 
stored in the image database, but are not necessarily physi- 
cally stored adjacent to the respective image in the image 
database. FIG. 2 shows the logical memory map 70 for the 
digital images stored in any of image databases 36 and 
40-40/1. 

As can be seen from FIG. 2, the logical memory map 70 
includes a number of entries 71-70/z, each for storing one 
digital image. Each entry (e.g., the entry 71) includes a 
content data field (e.g., the content data field 71a) and an 
image field (e.g., the image field lib). The content data field 
of each entry stores the content data of a digital image and 
the image field stores the image data of the digital image. 
The content data field of an entry is associated or linked to 
the image field of that entry. 

Referring back to FIG. 1, because the data service systems 
10 and 20-20/1 are connected together, a subscriber site 
connected to any of the data service systems 10 and 20-20/t 
can access and retrieve the digital images stored in any of the 
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image databases 36 and 40-40/1. Because each digital image 
stored in an image database of the databases 36 and 40-40/t 
contains both its pixel data and content data, the user at the 
subscriber site may only retrieve the content data, the pixel 

5 data, or both from the respective image database. This means 
when the image management system 100 of FIG. 3 is 
implemented in the data access network system 50, the 
search and retrieval are effective and efficient because only 
the content data of digital images need to be downloaded 
from one of the image databases 36 and 40-40/t. The image 
management system 100 will be described in more detail 
below, also in conjunction with FIGS. 3-6. 

Referring now to FIG. 3, the image management system 
100 includes a user interface 101 and an image database 103, 

i5 in addition to the content analyzer 102 and the search and 
retrieval engine 104. The image database 103 collectively 
represents the image databases 36 and 40-40n of FIG. 1 
when the image management system 100 is employed in the 
data access network system 50. 

20 In one embodiment, each of the components 101-102 and 
104 is a functional component and is implemented by 
software. In another embodiment, each of the components 
101-102 and 104 can be discrete hardware or firmware 
module. 

25 Referring to FIGS. 1 and 3, when the image management 
system 100 is implemented in the data access network 
system 50, the image database 103 and the search engine 
104 are implemented in each of the data service systems 
while the user interface 101 and the content analyzer 102 are 

30 implemented in a subscriber site (e.g., the subscriber site 11) 
in one embodiment. In another embodiment, all components 
except the image database 103 are implemented in a sub- 
scriber site. Alternatively, all components of the image 
management system 100 except the user interface 101 can 

3 5 be implemented in one of the data service systems (e.g., the 
data service system 10) of the data access network system 50 
while the user interface 101 is implemented in the subscriber 
site. 

Referring to FIG. 3, the content analyzer 102 is connected 

40 to the user interface 101 and the image database 103. The 
user interface 101 is also connected to the search and 
retrieval engine 104. The search and retrieval engine 104 is 
in turn connected to the image database 103. The search and 
retrieval engine 104 is also connected to the auto -annotation 

45 module 107. The content analyzer 102 is used to analyze 
each digital image it receives to extract content information 
from the image. The content analyzer 102 also stores the 
logical link (e.g., hyper-link) information between the con- 
tent data and the pixel data of an image into the image 

50 database 103 if the content data and the pixel data of the 
image are stored in separate locations or databases of the 
image database 103. FIG. 4 shows in more detail the content 
analyzer 102, which will be described in more detail below. 
Referring again to FIG. 3, the content analyzer 102 

55 receives the digital images 132 to be stored in the image 
database 103. The content analyzer 102 can also receive the 
digital image 131 from the user interface 101. The digital 
images 132 are fed to the content analyzer 102 from, for 
example, a storage means (not shown in FIG. 3). The digital 

60 images 132 can be acquired by scanning photos or pictures 
using a scanner, or by a digital camera. In one embodiment, 
the content analyzer 102 receives one digital image at one 
time. In an alternative embodiment, the content analyzer 102 
can receive and process a number of digital images at the 

65 same time. 

The user interface 101 is an interactive user interface that 
is part of an interactive window interface. The user interface 
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101 includes a window (not shown) that displays the digital In one embodiment, the search and retrieval engine 104 

image 131 or the content data 130. The image management retrieves the content data of all digital images from the 

system 100 uses the digital image 131 to search for any image database 103 into the engine 104. The search and 

similar images stored in the database 103. When the content retrieval engine 104 then compares the content data with the 

data 130 are used for the search instead of the digital image 5 content data of the digital image 131 or the content data 130 

131, the image management system 100 searches the image to output similarity scores between the two content data 

database 103 for any image having similar content data to based on me predetermined similarity measure. In one 

the content data 130. When the ^content data 130 are asso- cmbodiment) lhc similarity measure includes a set of thresh- 

ciated with the digital image 131, me image management olds and thc content data includc a set of va]ucs ^ 

thTrtabaL C 103 USe C ° 10 com P arisoD of * c corresponding values of two content data 

. , j ™ , , iL , yields differences between the values of the two content 

The user interface 101 also displays the search results J . . ~ . . , . 

, .% j* *i i • ... c a* u • i u a data - differences are then compared against their cor- 

(e.g., the digital images that are found to be similar based on .. 4 , , A . .„ r . b , , . , , , 

the similarity scores). In addition, the user interface 101 also responding thresholds^ simUanty score is obtained based 

includes a number of command or function buttons that can ° L n h ° w t ° f ^ c L d ^ c [ cnccs ™ lcss or ™ rc thc 

be activated using a user input device (e.g., mouse). 15 thresholds. The search and retrieval engine 104 can be any 

If the digital image 131 includes a number of faces, the kind of kn0WD ***** en e me ' 

user interface 101 allows the user to select one of the faces Id onc embodiment, the search and retrieval engine 104 

in the image 131 for processing. This allows the content retrieves both the content and pixel data of the similar 

analyzer 102 to compute the content data of the selected face images for display after a search. In another embodiment, 

or select the associated face meta-data from the content data 20 the search and retrieval engine 104 selectively retrieves the 

130. The search and retrieval engine 104 then searches for similar images in accordance with their similarity scores. In 

any images stored in the database 103 that contains faces a further embodiment, the search and retrieval engine 104 

similar to the selected face in the digital image 131. The face retrieves the pixel data of the similar images only when 

selection can be done by the user using the user input device display of these images is requested, 

in a conventional way. 25 In addition, the search and retrieval engine 104 may 

In addition, the user interface 101 also allows display of include a number of search and retrieval engines, each of 

multiple images and retrieval based on faces present in those which is located in one of the data service systems 10 and 

multiple images. In this case, the user interface 101 allows 20-20/t (FIG. 1). When this is the case, the search engine 

the user to select and combine faces from different images 104 allows retrieval of the pixel data of the similar images 

displayed for the search. For example, the user can, through stored in the local image database of the image database 103 

the user interface 101, select faces in different images and while allowing the retrieval of only the content data of the 

combine the selected faces together for a boolean search similar images stored in the external image databases of the 

(i.e., at least one, AND, or OR). image database 103 into the local data service system. 

Once the user selects the desired function or operation, the 35 When one of the digital images 132 is to be stored in the 

user interface 101 notifies the corresponding components for image database 103, the image is sent to the content analyzer 

the appropriate operation. The user interface 101 can be 102 to extract the content data from the image. The image 

implemented by any known interactive user interface tech- and its content data are linked together and stored in the 

no logy. image database 103. When the digital image 131 is received 

The user interface 101 receives the digital image 131 (or 40 in the user interface 101, the user interface 101 first sends the 

a number of images) to be searched. The digital images 131 image 131 to the content analyzer 102 to extract the content 

can be acquired by scanning photos or pictures using a data from the digital image 131. The content data are sent to 

scanner, or by a digital camera. This means that the user the search and retrieval engine 104 to be compared with the 

interface 101 is connected to an image capturing device content data stored in the image database 103. The refer- 

(e.g., a scanner or digital camera)(not shown in FIG. 4). The 45 ences to the content data matching the content data of the 

user interface 101 can also receive the content data 130 of digital image 131 are then sent back to the user interface 

an image. 101. The user can then decide whether to view those images 

The search and retrieval engine 104 is used to receive the associated with those content data. If the user interface 101 

content data of a digital image and uses the content data to receives the image content data 130 (i.e., content query) 

search the image database 103 for images with the same or 50 instead of the digital image 131, the user interface 130 

similar content data. As described above, the content data directly sends the content data 130 to the search and retrieval 

include face feature data, color data, texture data, shape and engine 104. 

position data, and/or text description data of the image. The image management system 100 also includes an 
In one embodiment, the search and retrieval engine 104 auto-annotation module 107 that is connected to the search 
uses the face feature data to search for images stored in the 55 and retrieval engine 104. The auto-annotation module 107 
image database 103 having similar face feature data. In associates personal information (e.g., name) of a first face 
another embodiment, the search and retrieval engine 104 within a first image already stored in the image database 103 
combines the face feature data with other kinds of the with a second image having a second face that is found to be 
content data (e.g., the color data, texture data, shape and similar to the first face. The faces are found similar because, 
position data, and/or text description data) to search for 60 for example, they have matching or similar face feature data, 
images stored in the image database 103. This allows linking of images having different "faces" of the 
In one embodiment, the search and retrieval engine 104 same person. After the auto-annotation module 107 links the 
only uses one kind feature data (e.g., the face feature data or personal information with the second image, the second 
the color and texture feature data) to search for the similar image may then be stored in the image database 103. 
images. In another embodiment, the search and retrieval 65 The image management system 100 simplifies the man- 
engine 104 combines the face feature data with all other agement of digital images, resulting a large set of content- 
feature data of the content data for the search. indexed images. It also provides efficient and effective 



6,o: 

9 

search and retrieval for digital images. This, for example, 
reduces the time of finding a desired image for printing or 
transmission to a remote site for viewing. In other words, the 
image management system 100 greatly simplifies the access 
to available images. In addition, the image management 
system 100 provides a virtual distributed image database. 

FIG. 4 shows the structure of the content analyzer 102. As 
can be seen from FIG. 4, the content analyzer 102 includes 
a face detection system 201, a face feature extraction system 
202, and an image content analysis system 203. In addition, 
the content analyzer 102 also includes a linking system 204 
that associates the extracted content data with the respective 
digital image. The linking system 204 provides the logical 
linking information between the content data of an image 
and the pixel data of the image. The logical linking infor- 
mation can be, for example, hyper-links. In this case, the 
content data and the pixel data of the image can be stored in 
different locations within a database, or even in different 
databases of the image database 103. Alternatively, the 
content analyzer 102 does not include the linking system 
204. In this case, the linking or association function is 
performed in the image database 103 where the image and 
its content data are stored. 

The face detection system 201 is connected to receive the 
digital images 132 and the digital image 131 via the user 
interface 101 of FIG. 3. The face detection system 201 
detects if the digital image received contains any face. If the 
image contains a number of faces, the detection system 201 
detects each of the faces. The face detection system 201 
detects the eye locations of a face if the image contains the 
face. The image data and the faces' eye location data are sent 
to the face feature extraction system 202 to compute the face 
feature data of the detected face. The face feature data are 
also referred to as face coefficient data. When multiple faces 
are detected, the face feature extraction system 202 extracts 
the face feature data of each of the faces and the respective 
data are associated with that face in that image. 

In one embodiment, the face detection system 201 is a 
neural network-based face detection system disclosed in a 
publication entitled HUMAN FACE DETECTION IN 
VISUAL SCENES, by H. Rowley (har@cs.cmu.edu), S. 
Baluja (baluja@cs.cmu.edu), and T. Kanade 
(tk@cs.cmu.edu) in November 1995. The publication is 
available from Carnegie Mellon University's Internet site at 
"www.ius.Cs.cmu.edu/IUS/har2/har/www/CM U-CS-95- 
158R/." Alternatively, other known face detection systems 
may be used for the face detection system 201. 

When the face detection system 201 is the neural 
network-based face detection system, the face detection 
system 201 detects if the image contains any face by first 
applying a set of neural network-based filters to the image. 
The filters examine each location in the image at several 
scales, looking for locations that might contain a face (i.e., 
looking for eye locations). The face detection system 201 
then uses an arbitrator to combine the filter outputs. The 
arbitrator is used to merge detections from individual filters 
and eliminate overlapping detections. Using the neural 
network-based face detection system as the face detection 
system 201 makes the face detection robust, relatively fast, 
and detect most faces. In addition, it allows the face detec- 
tion system 201 to detect different kinds of faces with 
different poses and lightings. 

Once a face is detected from an image, the image and the 
eye locations of the face are sent to the face feature extrac- 
tion system 202. In one embodiment, the face feature 
extraction system 202 is implemented by a face feature 
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extraction system described in U.S. Pat. No. 5,164,992, 
dated Nov. 17, 1992, and entitled FACE RECOGNITION 
SYSTEM. Alternatively, other known face feature extrac- 
tion systems may be used. 

5 The face feature extraction system 202 extracts the face 
feature data from the detected face. The face feature extrac- 
tion system 202 does this by first normalizing, warping (i.e., 
rotating and scaling), and masking the face to produce a 
reference face that has a set of predetermined eye locations. 

10 This means that through normalizing, warping, and masking 
of the detected face, the eyes of the detected face are mapped 
into the predetermined eye locations of the reference face. 
This is done by translating, scaling, and rotating the detected 
face and by adjusting the brightness and contrast of the 

15 detected face. In other words, this is to standardize or 
normalize the detected face. This is disclosed in the above- 
mentioned U.S. Pat. No. 5,164,992. 

Then the face feature extraction system 202 projects the 
standardized face into an Eigen face space to compute the 

20 face coefficients of the detected face. This is also disclosed 
in the above-mentioned U.S. Pat. No. 5,164,992. 

FIG. 5 shows the process of detecting faces in an image 
and extracting the face feature data from the detected face by 
the face detection system 201 and the face feature extraction 
system 202 of FIG. 4. As can be seen from FIG. 5, the steps 
301 and 302 are performed by the face detection system 201 
of FIG. 4 and the steps 303 and 304 are performed by the 
face feature extraction system 202 of FIG. 4. At the steps 

3Q 301-302, the face detection system 201 detects if the image 
contains a face and, if so, the eye locations of the face. At 
the step 303, the face feature extraction system 202 of FIG. 
4 performs the normalization, wrapping, and masking func- 
tions on the detected face. This essentially maps the detected 

35 face into a reference face such that the eyes of the detected 
face are mapped into the predetermined eye locations of the 
reference face. 

Referring back to FIG. 4, the digital image applied to the 
face detection system 201 is meanwhile also applied to the 

40 image content analysis system 203. The image content 
analysis system 203 analyzes the image and extracts visual 
content data from the digital image. The visual content data 
include color, texture, shape, and position data of the content 
data. The color and texture data indicate properties such as 

45 color percentages, color layout, color histogram, color 
moment, color coherence, texture compactness, texture 
contrast, texture directionality, texture periodicity, and tex- 
ture homogeneity. In one embodiment, the image content 
analysis system 203 is implemented by a QBIC(™) (Query 

50 By Image Content) system manufactured and sold by IBM 
Corporation of Armonk, New York. The QBIC(™) system 
can be downloaded from IBM's world wide web site at 
"wwwqbic.almaden.ibm.com," Alternatively, the image 
content analysis system 203 can be implemented by other 

55 known color and texture extraction systems. 

In one embodiment, the image content analysis system 
203 also receives text description data of the image from the 
user interface 101 (FIG. 3). The text description data are 
supplied by the user through the user interface 101. The 

60 content analysis system 203 adds the text description data 
into the content data. 

The image and its extracted content data from the image 
content analysis system 203 and the face feature extraction 
system 202 are then sent to the linking system 204 such that 

65 they can be associated with each other before being stored 
in the image database 103 of FIG. 3, or being sent to the 
search and retrieval engine 104. As described above, the 
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linking system 204 also provides the logical linking infor- 
mation of the content data and the pixel data of the image 
when they are stored at separate locations. 

FIG. 6 shows the process performed by the searching and 
retrieval system 104 of FIG. 3 for searching and retrieving 
the digital images from the image database 103. As can be 
seen from FIG. 6, at the step 401, the image database or 
databases from which the search is to be conducted are 
selected. At the step 402, the content data of all the digital 
images stored in the selected database or databases are 
retrieved. In one embodiment, the content data of an image 
also include a low resolution version of the image that 
functions as an icon of the image. Alternatively, the content 
data do not include such icons. 

At the step 403, the user-defined query content data (i.e., 
the content data 130 or the content data of the digital image 
131 of FIG. 3) are sent to the search and retrieval engine 104. 
Then the query content data are compared with the stored 
content data. Any stored content data matching or similar to 
the query content data are identified and their corresponding 
images retrieved at the step 405. Alternatively, the search 
engine 104 of FIG. 3 only sends the icon data of those 
matching content data to the user interface 101 for display. 
At step 406, it is determined if more queries are received for 
comparison. Alternatively, only the references to the match- 
ing images are sent to the user interface 101. In this case, the 
icon data or image data are sent later when display of these 
images is requested. If not, the process ends at the step 407. 
Otherwise, the steps 403-405 are repeated. 

In the foregoing specification, the invention has been 
described with reference to specific embodiments thereof. It 
will, however, be evident to those skilled in the art that 
various modifications and changes may be made thereto 
without departing from the broader spirit and scope of the 
invention. The specification and drawings are, accordingly, 
to be regarded in an illustrative rather than a restrictive 
sense. 

What is claimed is: 

1. A digital image management system, comprising: 
(A) a content analyzer that analyzes an input image to 
extract content data from the image, wherein the con- 
tent data include face feature data, color data, and 
texture data of the entire image, as well as shape and 
position data of objects within the image, wherein the 
content analyzer further comprises 
a neural network-based face detection system that 

detects if the image contains a face; 
a principle component analysis system that computes 
the face feature data from the detected face within 
the image by (1) standardizing the face and (2) 
projecting the standardized face into an Eigen face in 
order to compute the face feature data, wherein the 
computed face feature data are not a low resolution 
sub-image of the detected face; 
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an image content analysis system that extracts the color 
data, texture data of the image, and shape and 
position data of objects within the image; 

(B) an image database coupled to the content analyzer to 
5 store pixel data of each of a plurality of images and the 

content data of each of the images associated with that 
image; 

(C) a search engine coupled to the image database and the 
10 content analyzer to compare the content data of the 

images with the content data of the input image such 
that any image similar to the input image can be 
identified from the image database without retrieving 
the pixel data of the image from the image database, 
is wherein the search engine combines the face feature 
data of the input image with the color data, the texture 
data, and the shape and position data of the image to 
search the image database. 

2. The digital image management system of claim 1, 
20 wherein the neural network-based face detection system 

detects if the image contains a face by dividing the image 
into a plurality of small windows and determining whether 
each of the windows contains a face, wherein the image may 
^ includes a plurality of faces. 

3. The digital image management system of claim 1, 
wherein the principle component analysis system extracts 
the face feature data of the face by normalizing the face into 
a standardized face such that eyes of the standardized face 

30 lie at predetermined locations in a reference map, and then 
computing the face feature data corresponding to the stan- 
dardized face. 

4. The digital image management system of claim 3, 
wherein the principle component analysis system normal- 

35 izes the face into the standardized face by translating, 
scaling, and rotating the face and by adjusting brightness and 
contrast of the face. 

5. The digital image management system of claim 1, 
wherein the image content analysis system extracts color 

40 percentage, color layout, and texture of the image to extract 
the color and textual data. 

6. The digital image management system of claim 1, 
further comprising a user interface that displays the input 
image, wherein when the input image includes a plurality of 

45 faces, the user interface allows a user to select at least one 
of the faces within the image to be searched by the search 
engine. 

7. The digital image management system of claim 1, 
further comprising an auto-annotation module coupled to the 

50 search engine to associate personal information of a first 
face within a first image stored in the image database with 
a second image having a second face that is similar to the 
first face of the first image. 

***** 
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[57] ABSTRACT 

Apparatus and method are disclosed for selecting multime- 
dia information, such as video, audio, graphics and text 
residing on a plurality of Data Warehouses, relational data- 
base management systems (RDMS) or object-oriented data- 
base systems (ODBA) connected to the Internet or other 
network, and for linking the multimedia information across 
the Internet, or other network, to any phrase, word, sentence 
and paragraph of text; or numbers; or maps; charts, and 
tables; or still pictures and/or graphics; or moving pictures 
and/or graphics; or audio elements contained in documents 
on an Internet or intranet web site so that any viewer of a 
web site, or other network resource, can directly access 
updated information in the Data Warehouse or a database in 
real time. The apparatus and method each: (i) stores a 
plurality of predetermined authentication procedures (such 
as user names and passwords) to gain admittance to Data 
Warehouses or databases, (ii) stores the Universal Resource 
Locators of intranet and Internet addresses of a plurality of 
expert predetermined optimum databases or Data Ware- 
houses containing text, audio, video and graphic 
information, or multimedia information relating to the infor- 
mation on the web site or other network resource; (iii) stores 
a plurality of expert-predetermined optimum queries for use 
in the search engines of each of the pre-selected databases, 
each query representing a discrete searchable concept as 
expressed by a word, phrase, sentence or paragraph of text, 
or any other media such as audio and video on a web site, 
or other network resource; and (iv) presents to the user the 
results of a search of the Data Warehouse or database 
through a graphical user interface (GUI) which coordinates 
and correlates viewer selection criteria with the expert 
optimum remote database selection and queries. 

61 Claims, 17 Drawing Sheets 
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METHOD AND APPARATUS FOR 
SELECTIVELY AUGMENTING RETRIEVED 
TEXT, NUMBERS, MAPS, CHARTS, STILL 
PICTURES AND/OR GRAPHICS, MOVING 
PICTURES AND/OR GRAPHICS AND AUDIO 
INFORMATION FROM A NETWORK 
RESOURCE 

BACKGROUND OF THE INVENTION 

1. Field of the Invention 

The present invention relates to information retrieval, and 
the application and deployment architecture for such infor- 
mation retrieval. Specifically, the present invention concerns 
a multi-tier client/server model for record retrieval wherein 
optimum record retrieval from a database is achieved based 
on embedded expert judgments linked to words, phrases, 
sentences and paragraphs of text; or numbers; or maps, 
charts, and tables (including spread sheet; or still pictures 
and/or graphics; or moving pictures and/or graphics; or 
audio elements (hereinafter sometimes collectively referred 
to as the "links" or "Linked Terms," or when any one of the 
aforementioned elements are used singly, as the "link" or 
"Linked Term"), contained in documents on a network 
resource, such as a web site and incorporating an intuitive 
graphical user interface (GUI) to correlate through a plural- 
ity of frames the retrieved records with records from one 
remote database or a large collection of remote databases 
maintained by one company, called a Data Warehouse, plus 
a means to select various databases or Data Warehouses and 
a comprehensive selectable index of the linked embedded 
expertjudgments. 

2. Background Information 
"Pull" Technology 

A conventional information retrieval system includes a 
database of records, a processor for executing searches on 
the records, and application software that controls how the 
retrieval system, such as a database management system 
(DBMS), accepts the search queries, manages the search, 
and handles the search results. Generally, the database 
includes records such as text documents, financial or court 
records, medical files, personnel records, graphical data, 
technical information, audio and video files or various 
combinations of such data. Typically, a user enters a pass- 
word and client billing information, and then initiates the 
search by finding the appropriate database or groups of 
databases to search and formulating a proper query that is 
sent to the DBMS. This process is known as searching by 
pull technology. To effectively search and retrieve records 
from the database, the DBMS typically offers a limited 
variety of search operations, or query models, specifically 
designed to operate on the underlying records in the data- 
base. The query models are coordinated and executed by an 
application generally referred to as a search engine. For 
example, a document database, such as a database of court 
opinions, may be organized with each court opinion as a 
record with fields for the title of the case, jurisdiction, court 
and body text. A simple search engine may support a full text 
searching query model for all the text fields, individual field 
searching, such as searching by court or jurisdiction, and 
various Boolean search operations such as and, or, and not. 
More sophisticated search engines may support the follow- 
ing query models: 

1. nested Boolean or natural language searches; 

2. grammatical connectors that search for terms in a gram- 
matical relationship such as within the same sentence or 
paragraph (e.g., "/s", "/p", etc.); 
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3. proximity connectors that require search terms to appear 
within a specified number of terms of each other (e.g., 
"w/5"); 

4. exclusion terms ("BUT NOT'); 
5 5, weighted keyword terms; 

6, wildcards; 

7, specification of the order in which the database processes 
the search request (e.g., grouping words in parenthetical 
expressions); 

1(J 8. restriction of the search to certain fields, and formulation 
of a restricted search such as by date, subject, jurisdiction, 
title, etc.; and 
9. combination of the fields of search. 
In addition, large commercial database providers, such as 
BLOOMBERG, DIALOG, LEXIS/NEXIS and WESTLAW 

15 typically have thousands of individual databases. These 
large commercial database providers are Data Warehouses, 
which comprise an architecture and process where data are 
extracted from external information providers, then 
formatted, aggregated, and integrated into a read only data- 

20 base that is optimized for decision making. Users subscribe 
to the Data Warehouses by monthly or yearly subscription, 
and then typically pay stratified levels of hourly charges for 
access to certain databases, or groups of databases. 
Drawbacks of Pull Technology 

25 One limitation of existing information retrieval systems, 
especially among the commercial Data Warehouses, is the 
burden on the user to first enter client and billing information 
and passwords to gain access and initiate the search, and 
then formulate the search query. Typically, the subscription 

30 based commercial database services provide password 
administration and extensive catalogues, both in print and 
on-line, describing the content and scope of the databases 
offered, and in some cases, live assistance by telephone by 
reference librarians who assist the user to find the proper 

35 databases. However, the user must remember the password, 
and spend time finding the proper database by catalogue, 
on-line access, or phone, or else incur more expensive 
hourly charges searching through single databases or groups 
of databases for the appropriate database content and scope. 

40 A second limitation of pull technology is the formulation 
of the search query. To use the more powerful commercial 
Data Warehouses effectively, a user must be trained to use all 
of the aforementioned query models, and have sufficient 
knowledge of the topic to choose the appropriate keywords 

45 or natural language terms. The complexity of the search 
process compels the commercial Data Warehouses to offer 
training and keyword help to their subscribers by multiple 
publications that describe search tips; interactive software 
based training modules; account representatives who visit 

50 the user and train him or her; and customer service and 
reference librarians available by phone. 

A third limitation of pull technology concerns how it is 
employed on the World Wide Web area of the Internet 
("WWW") by such search engines as THE ELECTRIC 

55 LIBRARY, EXCITE!, FOUR ONE ONE (411), HOTBOT, 
INFOSEEK, LINKS TAR, LYCOS, MAGELLAN ALTA 
VISTA, OPEN TEXT INDEX, WEB CRAWLER, 
WWWWORM, and YAHOO!, just to name a few. These 
search engines' query models are beginning to approach the 

60 sophistication and complexity of those of the commercial 
database companies, but unlike the commercial databases, 
they offer minimal customer support. Another drawback of 
the Internet search engines, well documented in the com- 
puter business and popular press, is that their search engine 

65 algorithms cause multiple irrelevant responses to a query. 
Other drawbacks of Internet search engines employing pull 
technology include: 
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1. The great majority of the Internet search engines have no the search and obtain the results in three to five different 
control over the records in their database. Unlike the Data Warehouses, each of which are organized differently 
commercial Data Warehouses who have an ongoing rela- from one another and have different methods to enter search 
tionship with the content provider (usually by a license queries, and different query models. While pull technology 
agreement), and who carefully screen, cleanse and format 5 satisfies the demands for the breadth and depth of the search 
the information provided by their information providers, (since the user can formulate his or her own queries, and 
many Internet search engines sweep through the WWW make unlimited selections of databases to search) it is time 
periodically and automatically, and catalogue web sites as consuming, cumbersome and expensive because the user 
records in their databases. They also permit any web must find the appropriate query formulation and database or 
publisher to submit his or her web site as a record entry 10 databases within which to run the query, sometimes even in 
with little or no prior screening. different Data Warehouses. 

2. As a result of little or no screening, and absolutely no "Push" Technology 

contact with the information provider, Internet search In response to the flood of information facing the typical 

engines often provide search results that have multiple Internet user under the pull model, the complexity of the 

"dead ends," the result of links which are often moved or 15 query statements, and the well documented inability of the 

deleted after the search engines have catalogued them, Internet search engines to locate and deliver relevant 

Moreover, the web sites' authors can sometimes manipu- content, software companies developed software agents to 

late the words on their site and cause the Internet search push information to users. The push model is also known as 

engines to list their websites higher on the search engine's webcasting. 

relevancy lists than other web sites. 20 Under push, computers sift through large volumes of 

3. The search engines' databases include only a fraction of information, filtering, retrieving and then ranking in order of 
the Internet's content, and even then, the content may be importance articles of current interest. The user fills out a 
from dubious sources, or sources which are not updated "profile" (also called a "channer"), that defines a predefined 
frequently. area of interest or activates a filter. This, in turn, causes the 

4. Where the web sites include embedded search terms in 25 webcast search engine to search its own databases, or the 
links in documents to existing Internet search engines or databases of others, for content matching the profile or the 
current awareness "news" databases, since the words are filters submitted by the user. The user, in order to access the 
linked to the free Internet search engines discussed above, channels and have the content "pushed" to him or her, must 
the information retrieved, for reasons explained above, is download special client software which acts either indepen- 
not reliable and users often receive multiple irrelevant 30 dently of, or in conjunction with, the user's browser, 
responses. Words linked to the current awareness data- Alternatively, a user can access a dynamically generated web 
bases receive more useful information, but there is no page on the webcaster's server that lists the found articles. 
GUI correlating and synchronizing the records of multiple (An example of a dynamically generated web page is 
databases. Typically, those web sites pass authentication "Newspage Direct" by Individual, Inc.) 

information by the QUERY_STRING environment vari- 35 One early version of the Internet push model, developed 

able. Once placed on the command line by the browser, by PointCast Inc., clogged the network behind a company's 

the viewer can see all passwords and usemames in the employees' firewall when large numbers of the company's 

authentication argument. software agents pulled information from PointCast's servers 

The considerable logistical and practical drawbacks of on the Internet at or near the same time. PointCast later 

pull technology are illustrated in the following example of 40 alleviated this problem by providing remote servers that 

an investment banker who is responsible for buying bonds could operate behind a company's firewall and request and 

for an institutional investor, such as a bank or an insurance collect (or cache) information at once or at predetermined 

company. This hypothetical investment banker, based on an times from the PointCast severs on the Internet. These 

actual person, will be used at different points throughout this intermediate servers then pushed the information to 

patent application to illustrate and support the novelty and 45 employees, which effectively centralized the distribution of 

unobviousness of the present invention. information in the Information Services (IS) department. 

Every week, this investment banker must go before a As mentioned above, all push technology requires that 

board of executives at his bank and provide them with a list users compile a "profile" to detail their interests. The prior 

of bonds that he had examined and analyzed and recom- art of delivering the information obtained by the search 

mends to the bank to buy. In order to do his due diligence 50 engine pursuant to the profile is divided into three broad 

he must cover in his report five areas of research concerning categories: offline browsers; e-mail delivered content pro- 

the bond: 1) compare the bond price to other bond prices (the viders and information channels. 

Bond Comp arables); 2) obtain historical data concerning the The offline browsers typically operate by requiring a user 

bond and the company issuing the bond (the Historical to complete a profile with predetermined categories; auto- 

Data); 3) obtain the Securities and Exchange filings, such as 55 matically search the Internet for the information specified in 

lOK's, and lOQ's for the company issuing the bond (the the profile and download the materials to the user's hard 

SEC Filings); 4) obtain specific information from a wide drive for viewing at a later time when the user is off the 

variety of publications concerning the industry in which the Internet. This first category of products include: Freeloader 

company operates (the Industry Data); and 5) obtain infer- by Freeloader, Inc.; Smart Delivery by FirstFloor, Inc.; 

mation concerning the historical and anticipated perfor- 60 WebEx by Traveling Software, Inc.; WebRetriever by Folio 

mance of the company's stock (the Stock Data). Inc. and Web Whacker by ForeFront Group, Inc. 
Furthermore, he has to read various newsletters and white The second category of push products delivers the results 

papers issued by investment banks desiring to sell the bonds of searches performed pursuant to the user's profile directly 

to him, and which analyze the bonds using the same criteria to the user's e-mail box, and includes: Netscape's Inbox 

mentioned above. In order to collect the data, this invest- 65 Direct and Microsoft Mail. 

ment banker must log on and enter password and billing The third category of push products arranges the prede- 

information; find the appropriate databases; and formulate termined categories into "channels" and uses filters to allow 
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users to customize their news deliveries from a broad range 
of proprietary news sources. It is claimed that the results of 
the searches are pushed or "broadcast" in real time to the 
viewer. Examples of this type of service include: BackWeb 
by BackWeb, Inc.; Headliner by Lanacom, Inc.; Incisa by 5 
Wayfarer, Inc.; Intermind by Intermind, Inc.; PointCast by 
PointCast, Inc.; and Marimba by Marimba, Inc. However, 
since the retrieved data is first cached on the service pro- 
vider's server (e.g. PointCast's server), and then again on the 
companys' servers behind the firewall, the results of the 10 
search are not really "broadcast in real time." 

There is a fourth category of push products which do not 
fall neatly into any of the above three categories of delivery. 
Citizen 1 by Citizen 1 Software, Inc., is a human organized 
hierarchical listing of free Internet search engines. The user 15 
can then select a number of databases which fall under that 
category, and run several simultaneous queries in the data- 
bases. Digital Bindery by Digital Bindery Company allows 
users to "subscribe" to web pages as they browse. Once a 
subscriber, the user will automatically receive via e-mail any 20 
updates to the web pages to which the user subscribed 

Webcasting attempts to eliminate the inefficiencies of pull 
technology, namely the time consuming and unproductive 
hunt for information through Internet search engines. 
Instead of an open ended search through many databases 25 
linked to the web by various search engines, as is done under 
the pull model, push substitutes one central secure database 
which has collected either the content itself, or the links to 
the content. However, in spite of the name, push, the 
information provider does not drive the distribution of data. 30 
Instead, a client (in a client/server arrangement) contacts the 
information provider and requests the information. The 
client then downloads the information in the background, 
giving the impression that it is broadcast, when in fact, it is 
only automatically downloaded at a predetermined time. 35 

Shortcomings of "Push" Technology 

"Push" may be a satisfactory method for serving infor- 
mation to knowledge workers who depend on a constant 
stream of updated factual information served in narrow 
categories. Examples of these kinds of workers would be 40 
sales representatives who must find new prospects, staff in 
field offices who must be aware of sudden price changes, 
information managers who must distribute software 
upgrades and marketing professionals who must be aware of 
the new products released by the competition. 45 

However, there is a category of knowledge workers 
whose information needs are not properly satisfied by push 
technology. The hypothetical investment banker discussed 
above is an example of such a knowledge worker. These 
knowledge workers cannot use "filters" and "profiles" to 50 
provide the most relevant information since the information 
they need cannot easily fit into categories, but rather spans 
categories. These knowledge workers use information to 
solve problems that are rarely alike. They need information 
to solve a problem, but they do not know what they need day 55 
to day. 

This knowledge worker culls information and sparks 
creativity by comparisons and contrasts, juxtapositions, and 
induction and deduction, rather than by looking at raw news 
reports. The investment banker discussed above, usually 60 
does not know well in advance what industry or company he 
will be analyzing. He also does not always know where his 
research and analyses will take him, or what databases he 
will use. His decisions are tied into so many variables that 
exist in the marketplace that his information cannot be 65 
predetermined by a general form or profile. A further limi- 
tation of webcasting is that it has not struck the optimum 
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balance between burdening the viewer with a persistent 
stream of alerts versus alerting the viewer when new infor- 
mation has arrived. 

Moreover, since webcasting centralizes the development, 
control and the administration of "profiles" within an Infor- 
mation Services (IS) department, certain knowledge work- 
ers' information needs may not be satisfied by such central- 
ization. IS departments, already strapped for resources to 
manage mail servers, web servers, Lotus Notes servers and 
application servers, may not be capable of managing servers 
that maintain lists of user "profiles" and dispatch software 
agents into the World Wide Web (WWW). The push model 
works only if IS departments pro actively keep the profile 
lists current and advertise them internally. Furthermore, 
there may be enormous legal ramifications, as of yet not 
addressed, to companies downloading copyrighted material 
to their internal servers and redistributing it internally, 
especially if the push purveyor links to other websites or 
search engines without permission. See, "Legal Situation Is 
Confused on Web Content Protections,". New York Times, 
Jun. 9, 1997, at page D5. 

Finally, all the above examples of "push" technology, 
except for "Digital Binder," require the buying, installation, 
maintenance and updating of software by both the publisher 
and the user. 

In addition to the above-mentioned disadvantages, both 
the push and pull models fail to address the need to 
efficiently, inexpensively, and frequently augment web sites 
with current or historical data. According to the Mar. 11, 
1997 Wall Street Journal, in an article entitled A t Thousands 
of Web Sites, Time Stands Still: "Nearly five million pages of 
a total 30 million indexed by AltaVista on the Web haven't 
been updated at all since early 1996 .. . Some 424,000 pages 
haven't been refreshed since early 1995 — and 75,000 Web 
pages haven't been touched since before 1994." 

Therefore, it is desirable to dynamically augment a static 
web page containing text, audio, graphics, and/or video 
information on a network resource with Linked Terms 
connected to current awareness and/or historical records 
from expert pre-selected Data Warehouses or single 
databases, thereby saving the enormous labor and time costs 
involved in updating web pages. 

It is similarly desirable to permit users to choose and 
narrow their own search criteria through pull technology by 
clicking on Linked Terms in a written document, and still 
obtain the benefits of push technology by having current 
awareness and historical records pushed to update their 
selections without introducing new protocols or application 
programmers interfaces (API's) to operate. It is therefore 
desirable to provide a method and apparatus use of which 
does not encumber the user's or publisher's computer sys- 
tem in the following ways: 1) neither the user, nor the 
publisher has to buy, install, maintain or update software to 
use the invention; 2) use of the method and apparatus does 
not require large hard disk and memory allocations by the 
user; and 3) as a result of "2," use of the method and 
apparatus does not preclude using other push products 
simultaneously. This invention can work with any operating 
system that employs a browser, and can accommodate any 
binary data type, including FTP repositories, full Java 
applets and VRML, and any browser plug-in, such as 
Shockwave applications. Moreover, it can deliver informa- 
tion from a variety of sources, including from the Internet, 
company databases, group ware and intra- and extranets. 

Finally, given the almost exclusive use of current aware- 
ness and historical data on databases for research purposes 
in the prior art, the present invention is unique and unob- 
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vious because it is the only invention that updates Linked 
Terms in any written document, including web pages, with 
current and/or archived information from databases and 
Data Warehouses using a proprietary user interface and 
embedded expert judgment. Updating web pages and written 
content in this matter effectively transforms raw information 
into data which can support any point made in any written 
document. So, for example, if the document is used for 
marketing purposes, this invention would permit raw infor- 
mation to be used for marketing purposes, etc. 

It is also desirable to provide a method and apparatus, 
which, rather than seeking to identify records on a database 
whose characteristics exactly match what the user types into 
a query model, embody one or more kinds of expert judge- 
ment data for the purpose of selectively retrieving on 
demand the best fitting or most appropriate records in 
response to user data entry. Accordingly, it is desirable to 
provide a query architecture for an information retrieval 
method and apparatus that utilizes both pull and push 
technologies wherein knowledge workers can select their 
database resources based on the issue they must solve and 
current awareness or historical data can be pushed to them 
based upon embedded expert judgment based on the same 
issue once they have selected the database resources. 

It is further desired that the Linked Terms in any docu- 
ment be augmentative and allow for the efficient integration 
of embedded expert judgment that correlates a user's choice 
of a Linked Term with optimum data information judgments 
or designations to identify those data where the fit between 
the user's choice of a Linked Term and optimum data for that 
Linked Term is best. 

SUMMARY OF THE INVENTION 

Broadly stated, the present invention encompasses a 
method of dynamically augmenting the contents of at least 
one file of information on a first network resource, said file 
of information having at least one link, said method com- 
prising the steps of: creating at least one request correspond- 
ing to said at least one link; coupling at least one query 
argument with said at least one request; providing a data- 
base; sending said at least one request and said at least one 
query argument to said database; causing said database to 
search for at least one record that satisfies said at least one 
query argument; providing a display for viewing by a user; 
subdividing said display into a plurality of frames; display- 
ing said at least one record that satisfies said at least one 
query argument in at least a first frame of said plurality of 
frames; and providing a series of graphic symbols in at least 
a second frame of said plurality of frames for enabling the 
user to select one of a plurality of databases. 

In a preferred embodiment of the method of the present 
invention, at least one authentication argument is coupled to 
said at least one query argument and said at least one 
request. Furthermore, in a preferred embodiment, said at 
least one request, said at least one query argument and said 
at least one authentication argument are sent to said data- 
base. Also in a preferred embodiment, the method of the 
present invention further comprises subdividing the display 
into a second plurality of frames to replace said plurality of 
frames and providing a list of index terms in at least one 
frame of said second plurality of frames, where the index 
terms are for selecting said at least one link in said at least 
one file of information on said first network resource. 

In a preferred embodiment, said at least one request 
comprises a request header specifying: a) a purpose of the 
request; b) a network address of a second network resource 
to which said request header is applied; c) a file name for an 
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application that is stored on said second network resource 
and d) an argument that acts as a key in a table lookup on 
said second network resource; and wherein an application 
that executes on a second network resource correlates in said 

5 table lookup said key with one of a plurality of expert 
predetermined optimum values, each said expert predeter- 
mined value comprising a network address for a database, a 
query argument, and an authentication argument. In a pre- 
ferred embodiment, the key can be used to create a window 

10 display for viewing by a user, said window presenting the 
user with a menu of choices for further areas of research 
pertaining to said key. The user, by selecting one of said 
choices, causes said application that is executed on said 
second network resource to match a key, corresponding to 

a5 said one of said choices in a table lookup, with a request 
header comprising: a) a purpose of the request; b) a network 
address for a third network resource to which said request 
header is applied; c) a file name for an application that is 
stored on said third network resource; d) a query argument; 

2Q and e) an authentication argument; and causes said applica- 
tion on said second network resource to send said request 
header to said third network resource. 

In another embodiment of the present invention, at least 
one embedded application is sent from the second network 

25 resource to the browser. The embedded application performs 
one or more of the functions previously performed by the 
application that is executed on the second network resource 
in the first embodiment of the present invention. The embed- 
ded application is preferably an applet. 

30 The present invention also encompasses providing an 
apparatus for dynamically augmenting the contents of at 
least one file of information on a first network resource, said 
file of information having at least one link, said apparatus 
comprising: a browser having a display for viewing by a 

35 user; a second network resource coupled to said browser, 
wherein said browser sends at least one request correspond- 
ing to said at least one link to said second network resource, 
further wherein said second network resource couples at 
least one query argument with said at least one request, said 

40 second network resource further causing said browser to 
subdivide said display into a plurality of frames; and a 
database coupled to said second network resource, wherein 
said second network resource sends said at least one request 
and said at least one query argument to said database, said 

45 database comprising a search engine for searching for at 
least one record in said database that satisfies said at least 
one query argument; wherein said at least one record that 
satisfies said at least one query argument is displayed in at 
least a first frame of said plurality of frames and a series of 

50 graphic symbols are displayed in at least a second frame of 
said plurality of frames for enabling the user to select one of 
a plurality of databases. 

In a preferred embodiment, the display is subdivided into 
a second plurality of frames to replace said plurality of 

55 frames and a list of index terms are displayed in at least one 
frame of said second plurality of frames, said index terms for 
selecting said at least one link in said at least one file of 
information on said first network resource. Also in a pre- 
ferred embodiment, said second network resource couples at 

60 least one authentication argument to said at least one request 
and said at least one query argument, further wherein said 
second network resource sends said at least one request, said 
at least one query argument, and said at least one authenti- 
cation argument to said database. Said second network 

65 resource preferably further comprises a memory, said 
memory storing a table look up having a plurality of expert 
predetermined optimum values, each said expert predeter- 
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mined optimum value having a network address for a to which a claim of copyright protection is made. The owner 
database, a query argument, and an authentication argument, has no objection to the facsimile reproduction by anyone of 
further wherein said at least one request comprises an the patent disclosure, as it appears in the Patent and Trade- 
argument that acts as a key in said table lookup, said key mark Office patent files or records, but otherwise reserves all 
being correlated with one of said expert predetermined 5 copyrights and similar rights whatsoever, 
optimum values, The present invention overcomes the limitations of the 

Id another embodiment of the apparatus of the present P. rior * rt by providing an augmentative query architecture 

invention, at least one embedded application is sent from the "V" ^ «J e CTeaUon > addltl0n and subsequent integration 

second network resource to the browser. The embedded of ^bedded expert judgement and authentication informa- 

application that is sent to the browser performs one or more w Uon ^ a query submitted to an information retrieval 

of the functions that are performed by the application on the s y stem > together with an intuitive GUI designed to correlate 

second network resource. ^ rou E h * P lurali, y of fan"* ^ responses of the informa- 

,. , , . . , . . tion retrieval system; a series of graphic symbols for 

Accordingly one object of the present mvention is pro- enabljn ^ ^ (<) ^ Qne of a ^ of databases or 

viding a method and apparatus to cost-effectively and Data Warehouses and an index of terms for selecting a link 

dynamically update web pages contaimng text, audio ^ a &]& rf information which fc ^ referred to ^ ^ an 

graphics, and/or video data that are a part of a network mformation t latc or , file of data on , first n6(work 

resource wiUi Linked Terms connected to current awareness resource which is also referred to below as a Kvm or 

or histoncal records from expert prc-selected Data Ware- Document Server. The invention comprises an information 

houses or single databases, without undue waste of time and ^ temp , ate which maybe a fl , e of ^ £ Hypertext Markup 

a or ' Language (HTML), or other mark up language embodying 

Another object of the present invention is to permit text, audio, graphics, and/or video elements, containing 

network users to have access to a large number of electronic Linked Terms and posted in the preferred embodiment on a 

database providers, such as BLOOMBERG, DIALOG, HyperText Transport Protocol (HTTP) server (the "Docu- 

DOW JONES, LEXIS/NEXIS and WESTLAW, etc, with- 2$ men t Server"), which is connected to a network. The infor- 

out being limited to a particular proprietary graphical user mation template can be a document specifically prepared for 

interface (GUI), entering passwords or billing information publication on the World Wide Web (for example, a docu- 

or being trained to use the query models for each Data men t in Adobe's Portable Document Format (PDF)); or a 

Warehouse. newsletter; white paper; or other document which has been 

Yet another object of the present invention is to permit 30 printed, but converted into HTML, 

wide spread dissemination of information from databases The hypertext links in the information template contain 

and Data Warehouses without the cost and security problems the HTTP or other network protocol addresses to a second 

to the Data Warehouses of training users to use the system HTTP server on a computer network. The second server acts 

or issuing and administering a large number of passwords. a s a proxy server (the "Proxy Server") to both the client 

These and other objects and advantages of the invention 35 application and a third HTTP server on a computer network 

will become evident to those skilled in the art in view of the connected through a database interface to a Data Warehouse 

following description of the invention. or database containing multimedia information searchable 

by the database's or Data Warehouse's proprietary search 

BRIEF DESCRIPTION OF THE DRAWINGS engine (the "Database Server"). When a user clicks on the 

The present invention is illustrated by way of example 40 h yp i c 1 5 li ? k5 Wearing ™ window of tfae browser, be or she 

and not limitation in the following figures in which like be m amultaneous interacUve communication with both 

references indicate like elements and in which: * e ^ Server and the Database Server across the net " 

work. 

FIG. 1 is a block diagram of a typical digital computer « ' „ „ . .. 4 . iL t 

utilized by a preferred embodiment of the present invention. , ™ e ^ S 7 er "P 8 » ^puter application Out uses 

' . , ,. 45 gateway protocols, such as the Common Gateway Interface 

FIG. 2 illustrates a sunple client/server system of the prior fCGY). The application includes look-up tables that com- 

art * prise authentication data for access to, and the network 

FIG. 3 illustrates a more complex client/server system of addresses of a plurality of Database Servers. The application 

the prior art. will also contain in its look up tables queries (such as 

FIG. 4 is a block diagram of one embodiment of the 50 Boolean search terms, date and field restrictions and 

information retrieval system of the present invention. connectors) that have been formulated by expert judgment to 

FIG. 5 is a flowchart of a method of operating the return the optimum results from the Data Warehoused 

information retrieval system of the present invention. proprietary search engine, and any other necessary informa- 

itt^c r t o n m jii i rj-i tion to authenticate a user, gain access to the Database 

FIGS. 6, 7, 8, 9, 10 and 11 are examples of display screens „ . L ■ I n * « r L , 7 

* a \ \u a - .u c .i. 55 Server, and run a search m the Data Warehouse s search 

presented to the user during the operation of the process . ~ tl _ n ' ... 

outlined in FIG 5 engine. The CGI application on the Proxy Server will act as 

a bidirectional conduit between the Proxy Server and any 

FIG. 12 is a flowchart of the method of operating the application on the Database Server that can accept at runt- 

Infodex indexing process of the present mvention. irne some form of fnrp data (for example, standard input 

FIGS. 13, 14, and 15 are examples of display screens 60 (stdin) or environment variables such as QUERY_ 

presented to the user during the operation of the process STRING). In another embodiment of the invention, the 

outlined in FIG. 12. Proxy Server instead of using CGI can use embedded 

DFTA1T FD nPSPRTPTTOM OP thp Structured Query Language (SQL) commands to pass the 

DETAILED DEOTffi^ION OF THE qu£ry argument difectly tQ ^ DaU Warehouse fa yet 

65 another embodiment, instead of using CGI or embedded 

A portion of the disclosure of this patent document SQL, one can use object oriented libraries such as 

contains material that is subject to copyright protection and Microsoft's Open Database Connections or Sun/Intersolv's 
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Java DataBase Connectivity-Object Database Connectivity 162, which also can send graphic or video output to output 

(JDBC-ODBC) Bridge) to pass the authentication argument device 130. Network connection or modem 163, or some 

and the query argument. other output device may also communicate with CPU 140 

When the user clicks on any one of the hyperlinks across bus 150. Network connection or modem 163 may 

contained in the document on the Document Server, the CGI s commumcate with other networks, such as the Internet, 

application on the Proxy Server will automatically return a extrlDels ' f"?! 8 " P rocessin g ^terns 1«3 across 

f c e jx ' j i , . T ™/ r . communication line 182. 
set of framed windows, described in the HTML, to the . . x . A , . . r . r 
. . . . . _ . . c n u . * j • .u It is to be noted that the following discussion of various 
browser, to which certain information will be targeted in the , ,. A ,. , . * c „ 
r 1 „ „ ,. tl , ■ , embodiments discussed herein will refer specifically to a 
frames by the Proxy Server, or directly by the Database . - ... 4 , . * , 
o u u *u *u n o a Ji A * u o m series of routmes which are generated in a high-level pro- 
Server or by both the Proxy Server and the Database Server 10 . / T«-r.T • • , \ 
/ J gramming language (e.g., the PERL interpretive language) 

y " which is interpreted and/or executed in computer system 100 
As a result of the browser's framed windows, and the at ^n-ams. These further are used in conjunction with the 
client's simultaneous connection with the Document Server, browser and scrver ^fontc available from Netscape, 
Proxy Server and the Database Server, the user will be able Microsoft and other producers of graphical browsers that 
to interactively access a range of expert pre-selected indi- CO mmumcate through network protocols such as HTTP, as 
vidual databases or databases in Data Warehouses linked to describe d above. It is also to be noted, however, that the 
the Linked Terms appearing in the framed windows of the following methods and apparatus may be implemented in 
browser. special purpose hardware devices, such as discrete logic 
FIG. 1 is a block diagram of a typical prior art computer 2Q devices, large scale integrated circuits (LSPs), application- 
system 100 utilized by a preferred embodiment of the specific integrated circuits (ASIC's), or other specialized 
present invention. Computer system 100 may be a general hardware, or implemented by combinations of the computer 
purpose computer or a specially designed computer having components with other non-computer components, such as 
features or capabilities equivalent to those described below Microsoft's Web -TV, or reduced capacity computers, such 
in relation to computer system 100. Computer system 100 ^ as Sun Microsystems, Inc's ("Sun Microsystem's") Net- 
comprises an alphanumeric input device, such as a keyboard work computer, which can consist of a web browser, a 
120, coupled to a processing means, such as a Central network connection, 4 MB to 8 MB of memory and a display 
Processing Unit (CPU) 140, such as an Intel Pentium chip. screen. The description here has equal application to appa- 
The input device may also be another computer, or other ratus or programming languages having similar function, 
known input devices. An additional user input device 110 for 3Q FIG. 2 illustrates a simple client/server system of the prior 
communicating cursor direction or selection, such as a art j n wn i cn a single, bidirectional communication line 
mouse, trackball, stylus, motion pad, or cursor direction establishes a connection between a client and server across 
keys may be coupled to CPU 140. CPU 140 is coupled to a network. The client is an application program that estab- 
Read Only Memory (ROM) 142 and main memory 141, Ushes connections over a network for the purpose of sending 
which in a preferred embodiment comprises Random Access 35 requests and receiving responses. The server is an applica- 
Memory (RAM). Additionally, input/output device control- ti on p rogram that accepts connections in order to respond to 
ler 170 and a bus 150 are coupled to CPU 140. requests sent by the client. A connection is a transport-layer 
Main memory, which may include RAM or some other virtual circuit established between two application programs 
volatile storage device, is for storing information and for the purpose of communication. 

instructions to be executed by the CPU 140. Main memory 40 In FIG. 2, client 200 is coupled, by a bi-directional 
141 may also be used to store temporary variables or other connection 201 to server 202 (typically, a remote computer 
intermediate information during execution of instructions by system accessible over the Internet or other network) which 
CPU 140. ROM 142, which may be replaced by or used in can parse an Internet protocol, such as Hypertext Transfer 
conjunction with some static storage device, is coupled to Protocol (HTTP). Client 200 sends requests for information 
CPU 140 for storing static information and instructions 45 to server 202 by bi-directional connection 201. Server 202 
during processing by CPU 140. Computer system 100 also searches for the requested data in its files, finds and retrieves 
comprises hard disk 164, which is a data storage device that it, and then presents the data as server responses to the client 
communicates with CPU 140 across bus 150. Computer 200 via bi-directional connection 201. As server 202, in a 
readable removable media 172, such as magnetic or optical preferred embodiment, operates in an HTTP protocol, it is 
disk (including but not limited to magnetic tapes, laser disks, 50 also referred to as an HTTP document server. HTTP is a 
or CD-ROMs or DVDS), may be coupled to an input/output communications protocol that supports distributed collabo- 
device 171. Data is read from or written to the removable rative information systems over the Transmission Control 
media by the I/O device under the control of the I/O device Protocol/Internet Protocol (TCP/IP) packet based routing 
controller 170. These media may be used for storage of the system used by the Internet, including TCP/IP version 6.0. 
various files to be described herein including graphic and 55 It is to be noted that other transport layer application 
text files. programming interfaces (API's) such as the Component 
Computer system 100 may also comprise an output device Object Model (COM); Distributed COM (DCOM); IBM's 
130, such as a cathode ray tube (CRT) or liquid crystal System Object Model (SOM) and Distributed SOM 
display (LCD), coupled to the CPU 140 for displaying (DSOM) for networks and Java; Microsoft's ActiveX; Corn- 
information to a computer user. In addition, an audio output 60 mon Object Request Broker Architecture (CORBRA) (with 
device 180 which converts digital information to analog enhancements) and Sun Microsystem's Java's Remote 
information and delivers the output through headphones, Method Invocation (RMI) for the Internet may also be used 
speakers, or other well known audio mixing and storage in the present invention. 

devices such as magnetic tape may be coupled to the bus 150 HTTP has an open-ended set of methods that can be used 

through the audio adapter 161. Other devices such as graphic 65 to indicate the purpose and location of a request. These 

(or video) output devices 181 may also be coupled to the bus methods signal the purpose of a request by using terms such 

150 via a graphics adapter (or a video accelerator adapter) as Simple Mail Transport Protocol (SMTP); File Transport 
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Protocol (FTP); or HyperText Transport Protocol (HTTP). 
Other methods can use the Uniform Resource Identifier 
(URI); Uniform Resource Locator (URL); or Uniform 
Resource Name (URN) to indicate the network resource to 
which a method is to be applied. A network resource is a 5 
network data object or service that can be identified by a 
URI, URL or URN. An example of a URL is: http:// 
www.example.com/file.html, which provides the address of 
subdirectory "file. html" on the network resource "www. ex- 
ample .com". A network resource may also be a server, a 10 
database or Data Warehouse. 

HTTP can also be used as a generic protocol for commu- 
nicating with other Internet protocols such as SMTP, Net- 
work News Transport Protocol (NNTP), FTP, Gopher or 
Wide-Area Information Services (WAIS). An HTTP mes- 15 
sage consists of a structured sequence of octets (a set of eight 
bits) transmitted by the connection. 

FIG. 3 illustrates a more complex client/server system of 
the prior art. The system shown in FIG. 3 will be used to 
illustrate the flow of information in a typical academic and 20 
research oriented on-line service, such as LEXIS-NEXIS. In 
the case of LEXIS/NEXIS, remote users would pay a yearly 
subscription fee plus stratified hourly charges to access the 
Data Warehouse run by this on-line service. To access 
LEXIS/NEMS, a user dials in from a remote PC using client 25 
200, which includes LEXIS 1 proprietary software, and sends 
requests via the service's closed-access network 205. Mul- 
tiple front-end communication servers 210, which can 
handle more than 3,000 simultaneous sessions during peak 
business hours and that run and use proprietary applications 30 
and standard and non-standard transport protocols, would 
feed the queries through a database interface 220 to five 
large multiple virtual systems (MVS)-based operating sys- 
tem servers, which are collectively designated as Data 
Warehouse 230. LEXIS/NEXIS supplements their MVS 35 
systems with 120 UNIX-based servers to manage data. All 
other on-line services such as DIALOG, WESTLAW and 
BLOOMBERG have similar setups. LEMS/NEXIS is an 
example of a Data Warehouse, as that term is used herein. 

40 

During 1996, all of the on-line services, including the 
academic and research oriented ones, migrated in varying 
degrees to the Internet and the World Wide Web (WWW) or 
simply the "Web". It is anticipated that, in order to provide 
access to the WWW via the Internet transport protocol HTP, 45 
the major research and academic databases, as well as all 
new database on-line services, will have to integrate HTTP 
servers into their front-end communications servers, and 
convert parts of their data stores into HTML, or use inter- 
faces to convert documents into HTML on the fly. Corporate 5Q 
intranets are also switching to the HTTP protocol and will 
integrate some form of HTIP servers or HTML conversion 
"on the fly" to access their legacy databases. 

By adhering to the HTTP protocol, a standard is devel- 
oped that reduces and simplifies the variety of interfaces and 55 
gateways used by the on-line services and Data Warehouses 
to provide widespread access to their data stores. This, in 
turn, makes it possible to interpose an HTTP server as a 
proxy server between the client and the database without 
additional complicated gateways or interfaces. As used 60 
herein, a gateway is any application program that receives 
data from a browser or other HTTP server and converts it 
into a form the database can understand. An interface is a 
software application that interacts with a database or Data 
Warehouse. 65 

FIG. 4 is a block diagram of one embodiment of the 
information retrieval system of the present invention. The 
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informational retrieval system, shown in FIG. 4, comprises 
client 203 which is coupled to Document Server 202 and 
Proxy Server 207. Active within the client 203 is a first 
process, known as a browser 204, which establishes the 
connection, via the HTTP protocol with remote servers. A 
browser is an application which runs on a client and which 
can access a variety of servers providing information, 
including HTTP servers. Client 203 is coupled to Document 
Server 202 by bi-directional connection 201 through which 
client 203 sends requests for information (client requests) to 
and receives information from document server 202 as 
described in relation to the system shown in FIG. 2. Client 
203 is coupled to Proxy Server 207 by HTTP connection 
206. Client 203 and Proxy Server 207 interactively commu- 
nicate with each other using the functionality provided by 
HTTP. The WWW includes all the servers adhering to this 
standard which are accessible to clients via TCP/IP address- 
ing methods, such as the URL's. For example, communi- 
cation can be provided over an HTTP protocol used on a 
TCP/IP network. The client application and server may be 
coupled via Point to Point (PPP), or Serial Line Internet 
Protocol (SLIP) for dial up connectivity to the Internet, or by 
56 KBPS, ISDN, Frame Relay, T-l, T-2, T-3 or T-4 for high 
speed connectivity to the Internet. 

Proxy Server 207 is coupled to Database Server 211 by 
connection 208. Proxy Server 207 and Database Server 211 
interactively communicate with each other using the func- 
tionality provided on connection 208 by the Common Gate- 
way Interface (CGI) and HTTP via connection 208. The 
Document Server 202, Proxy Server 207 and Database 
Server 211 are typical HTTP servers equipped with varying 
degrees of memory and hard drive space. At a minimum, an 
example of a typical installation of each server would 
consist of a Sun Microsystems Netra workstation running a 
Solaris 2.5 operating system and employing 32-64 mega- 
bytes of memory and 2 gigabytes of hard drive space. The 
workstation can use the National Center for Supercomputing 
Applications (NCSA) HTTP daemon or some other compa- 
rable software such as Netscape's Enterprise Server as the 
server software. 

Database Server 211 is in turn coupled to Data Warehouse 
230 via Database Interface 220. Data Warehouse 230 may 
include one or more databases. Database Server 211 acts as 
a front-end communications server to data stored in Data 
Warehouse 230 or an individual database. 

Applications stored and executed on Proxy Server 207 
can use a Common Gateway Interface (CGI) protocol inter- 
face; a CGI protocol interface with vendor added extensions 
such Oracle's Web Request Broker; a non-CGI programmic 
object oriented communications protocols interface, such as 
Java Servlet application programmer's interface (API), Java 
applets; or non-CGI programmic server extensions, such as 
Microsoft's Internet Server API, or Netscape API, as a 
gateway interface to Data Warehouse 230 or an individual 
database. It is to be noted that, according to the present 
invention, one of the aforementioned applications on the 
Proxy Server, among other things: 1) correlates in a table 
lookup, a key sent by the browser 204 on client 203 to Proxy 
Server 207 with one of a plurality of expert predetermined 
values (each expert predetermined value comprising a net- 
work address for a database or Data Warehouse, a query 
argument, and an authentication argument); and 2) generates 
a second request header where the request header comprises 
a purpose of the request, a network address for a third 
network resource to which the request header is applied, a 
file name for an application that is stored and executed on 
the third network resource, an authentication argument and 
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a query argument), and, if required, modifies a record that 
satisfies the query argument. Therefore, the application on 
the Proxy Server effectively couples a query argument to a 
request sent by browser 204 on client 203 to Proxy Server 
207. Additionally, the application preferably couples at least 5 
one authentication argument to the query argument and the 
request. Additionally, the applications on the Proxy Server 
207 preferably cause browser 204 to subdivide its display 
into a plurality of frames, cause a record that satisfies the 
query argument to appear in the largest of the frames, cause 10 
a series of graphic symbols, e.g., buttons, to appear in a 
second frame for enabling a user to manually select one of 
a plurality of databases or Data Warehouses; cause an 
excerpt of text that includes a link in a file of information to 
appear in a third frame; and cause information associated 15 
with the source of the file of information to appear in a fourth 
frame. When the user manually selects one of a plurality of 
databases or Data Warehouses, the application correlates in 
a table lookup a key, sent by the browser 204 when the user 
manually selected the database or Data Warehouse, with one 20 
of a plurality of expert predetermined values, and sends a 
third request header to the third network resource, such as 
Database Server 211. Thus, the applications both send a 
request header in response to the selection of the link or a 
button and match a key corresponding to the link or button 25 
with a request header. Similarly, the applications send the 
request, the query argument and the authentication argument 
to the Database Server 211. 

FIG. 5 is a flowchart of the method of operating the 
information retrieval system of the present invention. At step 30 
249 of the process, the user, employing a browser 204 
running on a client 203 (shown in FIG. 4), sends a request 
via a network protocol such as HTTP for an HTML file to 
the Document Server 202 (shown in FIG. 4). A typical HTTP 
statement is in the form http://www.example .com/file .html. 35 
The HTML file "file. html" may be an HTML document that 
will have words, phrases, sentences and paragraphs, or 
graphics, video and audio elements, each symbolized by an 
argument abbreviating the name of the term or client (the 
"Argument Symbol"), The HTML document may be a 40 
document prepared for publication on the WWW (including 
a Web page), a newsletter, or a white paper or other 
document which has been printed but converted into HTML. 
In the example shown in FIG. 5, the Linked Term is the 
phrase "AUTOMOTIVE-RELATED INDUSTRY," and the 45 
initial default Argument Symbol that is assigned to it is 
"AR1". An HTTP network address in the form of http:// 
www.example.com/datasite.pl is employed, where "http" is 
the purpose of the request, "www.example.com" is the 
address of the Proxy Server and "datasite.pl" is the name of 50 
a CGI application on the Proxy Server 207 (shown in FIG. 
4). The Argument Symbol is added to the end of the HHP 
network address after a question mark as shown in FIG. 5 
where "AR1" follows the question mark, "?". The HTTP 
network address and the Argument Symbol, "AR1," consti- 55 
tute the request corresponding to the Linked Term, 
"AUTOMOTIVE-RELATED INDUSTRY." The request 
comprises a request header specifying a purpose of the 
request ("hup"), the network address of Proxy Server 207 to 
which the request header is applied ("www.example.com"), 60 
a file name of an application that is stored on Proxy Server 
207 ("datasite.pl"), and an argument that acts as a key in a 
table lookup and corresponds to the request ("AR1"). When 
the user clicks on the Linked Term, "AUTOMOTIVE- 
RELATED INDUSTRY", it is determined at step 251 65 
whether the user must be authenticated. Thereafter, if there 
is a need to authenticate the user, the user is authenticated, 
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in a first authentication process, by the Proxy Server at step 
252 before proceeding to step 253. Otherwise, the process is 
continued at step 253 without the first authentication. 
Thereafter, the browser passes the request, including the 
Argument Symbol, to the CGI application on the Proxy 
Server 207 (shown in FIG. 4). For example, if the term 
AUTOMOTIVE-RELATED INDUSTRY is linked to three 
separate Database Servers, there will be three Argument 
Symbols AR1, AR2 and AR3 assigned to the term where 
each Argument Symbol contains the separate network 
address of each one of the Database Servers. Although FIG. 
5 uses as an example Argument Symbols Al . . . A10, there 
is in reality no limit to the number of databases to which a 
Linked Term can be connected, and therefore FIG. 5 should 
not be construed as a limitation. 

The Argument Symbol is used as a key in a table lookup 
on the Proxy Server that is implemented as a hash table, 
associative array or a linked list. The table look up matches 
the key with the expert -predetermined optimum values for 
the Database Servers' network address and query. 

Thus, browser 204 sends a request to Proxy Server 207 
(shown in FIG. 4). The request comprises a request header 
which specifies the purpose of the request, the network 
address of the Proxy Server 207, the file name of an 
application that is stored on the Proxy Server 207, and an 
argument that acts as a key in a table lookup and corresponds 
to the Linked Term associated with the request. In a pre- 
ferred embodiment, the request, including the request 
header, is sent to the Proxy Server 207 via a collaborative 
information systems transmission protocols used on a net- 
work which include a combination of Hypertext Transfer 
Protocol (HTTP) and Transmission Control/Internet Proto- 
col (TCP/IP), including TCP/IP version 6.0. 

Alternatively, the argument that acts as a key or the key 
can be used to create a pop -up or floating window display for 
viewing by the user. The window display presents the user 
with a menu of choices for further areas of research per- 
taining to the key and, therefore, the Linked Terms. Each 
choice corresponds to an Argument Symbol. The user 
chooses one of the options presented, which will send an 
Argument Symbol to Proxy Server 207 that is used as a key 
in a table lookup on Proxy Server 207. The table lookup then 
matches the key with one of a plurality of expert- 
predetermined optimum values used to retrieve records from 
the Data Warehouse or database. Each expert predetermined 
optimum value includes a network address for the Database 
Server, a query argument, and an authentication argument. 
In other words, the user by selecting one of the choices 
causes an application that is executed on Proxy Server 207 
to match a key, corresponding to the selected choice in the 
table lookup, with a request header comprising a purpose of 
the request, a network address for a database network 
resource to which the request header is applied, a file name 
for an application that is stored on the database network 
resource, a query argument and an authentication argument. 
Additionally, the selection of the choice by the user causes 
the application that is executed on Proxy Server 207 to send 
the request header to the database network resource, e.g., 
Data Warehouse 230 or a database. 

Next, the CGI application is executed, as shown at 253. At 
step 254, the CGI application finds and sends to the browser 
the file which contains the HTML code that causes the 
browser to subdivide its main viewing window into a series 
of frames 262, 263, 264 and 265. Simultaneously, at step 
255, the CGI application correlates the Argument Symbol 
with the network address of the Database Server and the 
query argument chosen to be inserted in the database's or 
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Data Warehouse's search engine, and sends a request header 
to the Database Server 211 (shown in FIG. 4). An authen- 
tication argument (e.g., a user name and password) and a 
query argument are coupled to the request (which includes 
a request header) corresponding to the Linked Term selected 
by the user. The request header includes a purpose of the 
request (e.g., http), a network address of the network 
resource to which the request header is applied, a file name 
for an application on the network resource to which the 
request header is applied and alphanumeric arguments that 
contain the user ID; password and search query 255. (The 
authentication argument is used to access a database or Data 
Warehouse that may require authentication prior to allowing 
access to files or records in the database or Data Warehouse.) 
Thereafter, the request header, the authentication argument 
and the query argument are sent to the database or Data 
Warehouse. In another embodiment, the request header, the 
authentication argument and the query argument are sent to 
the Database Server 211 associated with the database or 
Data Warehouse. Thereafter, the Database Server 211 parses 
the request header and sends the authentication argument 
and the query argument directly to the database or Data 
Warehouse. If the Database Server performs the 
authentication, then just the query argument will be sent to 
the database or Data Warehouse. The Database Server takes 
the query argument and passes it through the Database 
Interface 220 (shown in FIG. 4) to the Data Warehouse 257. 
The Data Warehouse search engines locate the records and 
send them back through the Database Interface 220 (shown 
in FIG. 4) and the Database Server 211 (shown in FIG. 4) to 
the (e.g., datasite.pl) on the Proxy Server for further 
processing, which will be described below. 

FIG. 6, shows the resulting view in the browser as a 
consequence of the Document Server's response to a user's 
request for a document. Browser window 300 is the initial 
screen of the document sent by the Document Server 202 
(shown in FIG. 4). Browser window 301 shows the same 
document after the user scrolls down to a Linked Term 303 
he or she wants to see. In one embodiment of the invention, 
such as that shown in FIG. 6, the hyperlinked term can have 
hyperlinked symbols 305 following immediately after the 
term which indicate to the reader the properties of the media 
linked to the Linked Term. For example, a "speaker" symbol 
will indicate audio content, a "film strip" segment symbol 
will indicate video and a "document" symbol will indicate 
text. These symbols could appear singly or in groups 
depending on the nature and properties of the content 
connected. 

FIG. 7 shows a view of the browser window after it has 
been divided into four frames. When the user clicks on the 
Linked Term, the browser sends a request to the Proxy 
Server. All or only some of the frames may be scrollable in 
either an up and down direction, or in a side-to-side direc- 
tion. The Proxy Server responds by outputting HTML 
<FRAMESET> and <FRAME> elements that are described 
in the CGI application on the Proxy Server causing the 
browser's windows to subdivide into four frames 400, 401, 
402, and 403 and 254, 262, 263, 264 and 265 (shown in FIG. 
5). It is to be noted that the present invention is not limited 
to subdividing the browser's window into four frames, as 
shown in FIG. 7. Therefore, in the other embodiments, the 
browser's window may be divided into any number of 
frames arranged in any desired way. 

For instance, as a result of clicking on the link 
"AUTOMOTIVE-RELATED INDUSTRY", the browser 
sends the request header http://www.example.com/ 
datasite.pl?ARl to the Proxy Server located at the network 
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address "www.example.com". The Proxy Server parses the 
incoming request, and locates the CGI application "data- 
site .pi". The Proxy Server then executes "datasite.pl" and 
outputs HTML <FRAMESET> and <FRAMES> element to 
5 the browser 254 (shown in FIG. 5), causing the browser to 
parse the HTML elements. As the browser parses the HTML 
elements, it causes the main viewing window in the browser 
to subdivide into four frames 400, 401, 402 and 403. The 
characteristics of the frames' functionality are as follows: 
10 Each frame can load a network address independently of 
the other frames; 
Each frame can be given a specific name (using the 
HTML "NAME" tag), allowing it to be targeted by 
other request headers; 
s Each frame can resize itself dynamically in response to 
changes in the size of its visible area, or it can be set to 
disallow dynamic resizing or manual resizing by a 
viewer; and 

The frames can either be standard fixed frames or floating 
frames. 

20 Each frame may be set to scroll or not to scroll. 

In addition to the frames, the HTML code sent to the 
client specifies text, audio, graphics or video files (or 
documents) or some combination thereof, to be pulled in 
from other directories on the Proxy Server, and/or from other 

25 remote servers across the Internet or within an intranet, and 
into predetermined nested frames on the client application. 
The HTML code in the CGI application on the Proxy Server 
which calls for the documents from the Database HTTP 
Server can specify the frames to which the file would be 

30 loaded by using the TARGET element and specifying the 
attribute given to the frame's NAME element. 

For example, when the HTML code sent by the CGI 
application on the Proxy Server loads into the browser, it 
may contain a FRAMESET tag, which is nested within a 

35 second FRAMESET tag, which is further nested within a 
third FRAMESET tag. 

Each FRAMESET tag is placed in the space that would be 
used for the corresponding frame if it had been a FRAME 
tag instead of a nested FRAMESET When the browser 

40 parses the HTML code from the Proxy Server, the browser 
will first divide the browser's window into two separate 
frames in a 78% to 22% ratio (the 78% window is 400 and 
the 22% window is the sum of the areas of 401, 402 and 
403). Next, the browser will subdivide the 22% window into 

45 a 47% to 53% ratio (the 53% window is 402 and the 47% 
window is the sum of the areas of 401 and 403). Finally, the 
47% window is further subdivided into a 72% to 28% ratio 
(the 72% window is 401 and the 28% window is 403). 
FIG. 8 shows frames 1 to 4 of the browser's window filled 

50 with multimedia content received concurrently from both 
the Proxy Server and the Database Server 258, 259, 260 and 
261 (shown in FIG. 5). Frame 1 500 is identified with a 
named attribute of the FRAME element such as NAME= 
frame 1, where "frame 1" is the attribute of the element 

55 "name". In the present invention, the FRAME element 
located within the CGI application ("datasite.pl") on the 
Proxy Server holds the SRC element (see example below) 
for the hyperlink, and passes to the Database Server the 
authentication argument (password and user ID) and the 

60 query argument for the Data Warehouse's search engine 
through a QUERY_STRING environment variable annexed 
after the "?/' The NAME attribute refers to the name of a 
particular frame (frame 1, frame 2, etc.), and the HTML 
document referred to in the SRC element is automatically 

65 loaded into that named frame. So, the results of the search 
called for by the SRC element below would be loaded into 
frame 1 500. 
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NAME-"f rame4' 
MAR GINHEI GHT=" 1 ' 



The FRAME elements within the nested FRAMESETS 
located within datasite.pl look like this: 
<FRAMESET ROWS-"78%, 22%"> 
<FRAME 

SRC="http://www.database. com/directory 1/CGI.pl? 

userlD-X & password-Y & SEARCH-Z 
<FRAMESET ROWS="47%, 53%"> 
<FRAMESET COLSo"72%, 28%"><FRAME 
SRC«"upl-a 1-1. html" NAME = "f r amc2" 

MARGINWIDTH=" 1" MARGINHEIGHT="1" 

SCROLUNGo"NO"> 
<FRAME 

SRC-"cibcl2-l.htmt 

MARGINWIDTH=" 1" 

SCROLLING="NO"></FRAMESET> 
< FRAME 

SR Co "Ink 1-2-1 .html" NAME = "f rame3" 
MARGINWIDTH«"1" MARGINHEIGHT="1" 
SCROLLING-"AUTO"></FRAMESET> 

<NOFRAMES> 
where "www.database.oom" is the URL of the database; 
"directory 1" is the directory on the server where the CGI 
application is located; "CGI.pl" is the Data Warehouse's 
CGI application located on the Database Server and X, Y 
and Z are the alphanumeric arguments that contain the user 
ID, password and search query, respectively, and which are 
passed to CGI.pl by the QUERY_STRING environment 
variable. The contents called for by the SRC hyperlink 
"www.database.comare automatically loaded into frame 1 
500; 261 and 262 (shown in FIG. 5). Likewise, the contents 
called for by the SRC hyperlink upl-al-l.html would be 
loaded in frame 2 501; 258 and 263 (shown in FIG. 5); the 
contents called for by the SRC hyperlink cibcl2~l.html 
would be loaded in frame 4 503; 260 and 265 (shown in FIG. 
5) and the contents called for by SRC hyperlink Inkl- 
2-1 .html would be loaded in frame 3 502; 259 and 264 
(shown in FIG. 5). 

It is to be noted that a second environment variable, such 
as PATH_INFO, can be used to pass data to the CGI.pl 
application. Furthermore, one can also use METHOD - 
POST, and pass the data by stdin and stdout in the same 
fashion. Since the CGI application (datasite.pl), and not the 
browser, makes the request to the Database Server, the user 
will not see either the passwords or user ID's on the 
command line of the browser if the QUERY_STRING 
environment variable is used. 

A second embodiment of the invention could include 
vendor extensions for faster CGI access, for example Ora- 
cle's Web Request Broker, or non-CGI programmic server 
extensions such as Microsoft's Internet Server API (IS API) 
and the Netscape API (NSAPI). 

In a third embodiment of the present invention, the CGI 
application may be replaced as a means to pass data by 
employing a non CGI programmic object oriented commu- 
nications protocol, such as Java Servlet API, in an applica- 
tion performing the table look-up of the that is executed on 
the Proxy Server, and have the Java Servlet API communi- 
cate data to a Java-enabled Database Server and a Java 
enabled browser. Such an embodiment would avoid the 
slowness of passing data through stdin and stdout, ot 
QUERY_STRING environment variables as is done by the 
CGI application, thereby speeding up the data transfer 
process by allowing the application performing the table 
look-up on the Proxy Server to run continuously instead of 
stopping and starting on demand as a CGI application would 
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do, and permitting the server's memory to cache data. The 
Java Servlet API also permits the servers and client to 
establish end to end (browser to Proxy Server to Database 
Server and back) channel security through Secure Sockets 
Layer (SSL) or Secure HyperText Transport Protocol 
(S-HTTP). It would also encrypt all data passing from the 
client to the Proxy Server and from the Proxy Server to the 
Database Server. 

A fourth embodiment of the invention would use a non 
CGI programmic object oriented communications protocol 
employing embedded applications, such as Sun Microsys- 
tem's Java applets or Microsoft's Active X, containing 
authentication arguments and query arguments for all the 
selected databases connected to the document. These 
embedded applications can be encrypted. For example, the 
applets would be downloaded from the Proxy Server to the 
browser, which would permit the browser to independently 
gain access to the various databases and Data Warehouses 
without continuously using the resources of the Proxy 
Server. Presently this is not possible with Java applets since 
there is a restriction for security reasons on the functionality 
of the applet. Presently, applets can only communicate to the 
server from where they were received, which in this case 
would be the Proxy Server. Applets cannot presently connect 
to other servers. Also, applets cannot inspect or change files 
on the local hard drive or spawn other programs, including 
other applets. However, these limitations are expected to be 
lifted once certificate based public-key cryptographic sys- 
tems and digital signature based systems are perfected. 

The fourth method and its corresponding apparatus oper- 
ate as follows. A browser sends a request (which is in the 
form of a request header), corresponding to at least one link 
in a file of information on the Document Server to the Proxy 
Server. The Proxy Server, in response to receiving the 
request, executes an application and sends at least one 
embedded application, preferably encrypted, to the browser. 
In one embodiment, the embedded application is Sun Micro- 
system's Java applet. In a second embodiment, the embed- 
ded application is Microsoft's Active X. The encrypted 
embedded application is executed on the browser and 
couples an authentication argument and a query argument 
with the request header. Thereafter, the browser sends the 
request header, the query argument and the authentication 
argument to the Database Server. It is to be noted that the 
authentication argument need only be coupled to the request 
header and sent to the Database Server if the Database 
Server requires authentication for providing access to files in 
the database or Data Warehouse. The browser also causes 
the database or Data Warehouse to search for records that 
satisfy the query argument. Simultaneously, the embedded 
application that is executed on the browser causes the 
browser to subdivide its display window into four frames. 
Furthermore, the embedded application that is executed on 
the browser displays at least one record that satisfies the 
query argument in the largest of the four frames. The 
embedded application also causes a series of graphic sym- 
bols (e.g, buttons corresponding to a plurality of arguments) 
for selecting a plurality of databases to appear in at least a 
second of said four frames for enabling the user to select one 
of a plurality of databases. In a preferred embodiment, the 
embedded application that is executed on the browser, 
further causes the browser to subdivide its display window 
into a second set of frames to replace the four frames. The 
embedded applications causes a list of index terms to be 
displayed in at least one of the frames in the second set of 
frames. The index terms are for selecting the Linked Term 
in the file of information on the Document Server. 
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Id one embodiment, the above method further involves and, thereafter sends the record to the browser. The embed - 

determining whether a first authentication of a user is needed ded application that is executed on the browser causes the 

for sending the request to the Proxy Server; and authenti- record that satisfies the query argument to appear in the 

eating the user if such authentication is required. largest of the four frames created on the browser's window. 

In a preferred embodiment, the request header includes an 5 The embedded application executing on the browser also 

Argument Symbol that acts as a key in a table lookup that causes a plurality of buttons corresponding to a plurality of 

is implemented as a hash table, associative array or a linked arguments for selecting a plurality of databases to appear in 

list. Furthermore, the embedded application that is executed the second of the four frames on the browser's window (i.e., 

on the browser correlates in a table lookup the key with one the means for selecting a plurality of databases). These 

of a plurality of expert predetermined optimum values, 10 buttons include arguments that act as keys in the table 

where each expert predetermined value includes a network lookup with all the keys corresponding to a Linked Term in 

address for a database or Data Warehouse, a query argument, the aforementioned file of information on the Document 

and an authentication argument. Alternatively, the key can Server. The user by clicking on one of the buttons causes the 

be used to create a pop-up or floating window display for embedded application that is executed on the browser to 

viewing by the user. The window display presents the user 15 match the key corresponding to clicked button and causes 

with a menu of choices for further areas of research per- the embedded application to generate a second request 

taining to the Linked Terms where each choice corresponds header that includes the following information: a) a purpose 

to an Argument Symbol. The user chooses one of the options of the request; b) a network address for a database network 

presented, which will send an argument that is used as a key resource (e.g., the Database Server 211) (shown in FIG. 4) 

in a table lookup in an embedded application that is executed 20 to which the second request header is applied; c) a file name 

on the browser. The table lookup then matches the key with for an application stored on the Database Server; d) a query 

one of a plurality of expert-predetermined optimum values argument; and e) an authentication argument. Additionally, 

used to retrieve records from the Data Warehouse or data- the embedded application that is executed on the browser 

base. Each expert predetermined optimum value includes a causes an excerpt of text that includes the selected link in the 

network address for the Database Server, a query argument, 25 file of information to appear in the third one of the four 

and an authentication argument. In other words, the user by frames on the browser's window. Finally, the embedded 

selecting one of the choices causes an embedded application application executing on the browser causes information 

executing on browser 204 (shown in FIG. 4) to match a key, associated with the source of the file of information to 

corresponding to the selected choice in the table lookup, appear in the fourth of the four frames in the browser's 

with a request header comprising a purpose of the request, 30 window. 

a network address for a database network resource to which It is to be noted that the embedded applications sent to the 

the request header is applied, a file name for an application browser depend on the type of functions that one desires to 

on the database network resource, a query argument and an transfer from the Proxy Server to the browser. Thus, at one 

authentication argument. Additionally, the selection of the extreme, one or more embedded applications are sent to the 

choice by the user causes the embedded application execut- 35 browser to allow it to perform all the functions that would 

ing on browser 204 to send the request header to the otherwise be performed by the applications on the Proxy 

Database Server 211 which passes it to the Data Warehouse Server. At the other extreme no embedded applications are 

230 or a database through the Database Interface 220 (all sent to the browser, in which case all the functions that are 

shown in FIG. 4). performed by the applications on the Proxy Server in the 

In one embodiment, the expert predetermined optimum 40 earlier described embodiments (without the transfer of 

values and the keys are stored on the Proxy Server and are embedded applications from the Proxy Server to the 

sent to the browser in response to a request to that effect by browser), continue to be performed by the applications on 

the browser. Once the browser receives the data, it executes the Proxy Server. In such an embodiment, the applications 

the embedded application and matches one of the expert on the Proxy Server would perform the functions performed 

predetermined optimum values with a key in the table 45 on the browser by the execution of the embedded application 

lookup. (s). It is also to be noted that the range of functions, 

In a preferred embodiment, the embedded application transferred from the second network resource (i.e., the Proxy 

executing on the browser sends a request header that Server) to the browser, between the above two extremes is 

includes the following: a) a purpose of the request; b) a also covered within the scope of the present invention. Thus, 

network address for a Database Server, a database or Data 50 in one embodiment of the present invention, some but not all 

Warehouse to which the request header is applied; c) a file of the functions that would otherwise be performed by the . 

name for an application stored on the Database Server; d) a applications on the Proxy Server are served by the browser, 

query argument; and e) an authentication argument. The second FRAME element automatically loads the 

The Database Server, in response to receiving the request contents of the first of a series of HTML files into frame 2 

header, the authentication argument and the query argument, 55 501. These HTML files contain the graphic images of button 

authenticates the user, and passes the query argument to the bars, with each button bar linked in its HREF statement to 

Data Warehouse 230 or a database through the Database a specific network address of a network resource, such as a 

Interface 220 (all shown in FIG. 4). The database or Data Database Server, a database or a Data Warehouse. These 

Warehouse executes a search and returns to the browser button bars alternatively appear in frame 2 501 as "pressed" 

records that satisfy the query argument. In another 60 or "down", or not pressed or "up," every time a viewer 

embodiment, a database or Data Warehouse directly receives presses a button bar. Such actions can be accomplished in 

a request, an authentication argument and a query argument, HTML code, or C++, ActiveX, Java, JavaScript, Visual 

authenticates the user, executes a search and returns to the Basic computer programs, or other programming languages 

browser records that satisfy the query argument. In one 268, 263 (shown in FIG. 5). 

embodiment, an application that is executed on the Proxy 65 Of course, it may be appreciated by someone familiar 

Server or the embedded application that is executed on the with the art that any graphic image, or selection process or 

browser modifies the record that satisfies the query argument scheme may be implemented as long as it shows the viewer 
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what databases or Data Warehouses have been selected and 
which ones have not. When a viewer clicks on a button bar 
linked via an HTTP address to a remote database or Data 
Warehouse, the HTML file associated with the button bar 
causes the viewer's browser to make a request for a record 5 
from the specified database or Data Warehouse in the same 
manner as described above for the initial QUERY-STRING 
request, with the results of the request displayed in frame 1 
500. At the same time, the button bar that the viewer has 
clicked appears in frame 2 501 as depressed, with the 10 
remaining button bars appearing not pressed or up. The same 
process described above is repeated for the other remaining 
button bars, each time a viewer presses a button bar. 

Simultaneously with the initial loading of content in 
frame 1 500 and frame 2 501, an HTML file located on the 15 
Proxy Server, containing the logos of the corporate firm 
sponsoring the access to the updated information, and the 
firm which has arranged the access to the database, or any 
other pertinent corporate logo or information appears in 
frame 4 503 . 20 

Finally, also simultaneously with the initial loading of 
content in frame 1 500, frame 2 501, and frame 4 503, the 
surrounding 15 or so words to the text which the viewer 
initially saw and clicked on will appear in frame 3 502. The 
same Linked Terms in area 303 of FIG. 6 will be highlighted 25 
in area 504 of FIG. 8. This frame and link provides the user 
when he or she clicks on the link, with a means to view the 
entire document, automatically scrolled to the place where 
the link appears from the Document Server, as was illus- 
trated in FIG. 6. 30 

It is to be noted when the Database Server 211 returns 
either the initial menu of results to.datasite.pl on the Proxy 
Server or the final document that the user chose from the 
menu to datasite.pl on the Proxy Server (FIG. 10; 500 and 
FIG. 11), datasite.pl can strip away certain predetermined 35 
HTML tags from the document. This facility would be 
useful, for example, if specific banners or links to other areas 
of the Data Warehouse need to be disabled before it reaches 
the user's browser. 

The arrangement of frames allows the viewer to view 40 
simultaneously the results of his or her search in frame 1 
500, determine via frame 2 501 which button was depressed 
and which database has been selected, observe the corporate 
sponsor of the service in frame 4 503, and view the sur- 
rounding 25 to 30 words surrounding the highlighted text in 45 
frame 3 502. By clicking on the highlighted selection in 
frame 3 504, the user may return to the original full text of 
the document he or she was viewing and automatically scroll 
to the exact place in the full text document which the Linked 
Term occurs. 50 

FIG. 9 illustrates a means to enlarge the viewing area of 
frame 1 500. This may be accomplished through a series of 
small graphic buttons 601, which when clicked, will cause 
a window containing only frames 1 500, frame 2 501 and 
frame 4 503, without frame 3 502 to appear, as shown in 55 
display 602. By clicking on another button in Frame 4 603, 
the viewer may cause frame 3 502 to reappear. Other buttons 
or graphical devices on the display window may also be used 
to increase and/or decrease the viewing area of any frame on 
the display window in a matter well known to those skilled 60 
in the art. 

FIG. 10 illustrates the display screen after the user 
invokes the expert embedded judgment and the database or 
Data Warehouse returns the results. The user is presented 
with a menu of choices with a brief summary attached 65 
corresponding to a selection of text, audio, graphics or video 
files or documents from which he or she can choose. The 
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Data Warehouse or database organizes the returned records 
in a menu format. The user may adjust the screen in the 
manners described above using the graphic buttons 601 if he 
or she wishes, or choose a text, audio, graphics, or video 
document or file from area 500 of the main screen, one such 
selection being 604. The user may also return to the origi- 
nating document 502 by clicking on the highlighted Linked 
Term 504. The user also has the option of choosing another 
Data Warehouse or database by clicking on any one of the 
buttons in frame 2 501. Once the user makes his or her 
choice, the Data Warehouse or database will deliver the 
document corresponding to the choice in frame 1 500. 

FIG. 11 illustrates the browser's window including the 
"Infodex" feature of the preferred embodiment of the 
present invention. An "Infodex" button may be situated 
either in frame 2 700, or in some other suitable area of the 
GUI. A second button may be situated on the main document 
itself. In the embodiment shown in FIG. 6, that button 304 
is located just under the masthead of the newsletter. By 
pressing the Infodex button, a viewer activates a link to the 
Proxy Server and makes a request through his or her browser 
for an HTML document containing two equal frames. 

FIG. 12 is a flowchart of the method of operating the 
"Infodex" indexing process of the present invention. At step 
750 in FIG. 12, the user chooses the Infodex button. By 
clicking on the button, the user passes an argument by the 
QUERY_STRING HTTP protocol to the CGI application, 
infodex.pl on the Proxy Server. At step 751, the Proxy 
Server executes the CGI application infodex.pl. As a result, 
at step 752 an HTML document containing an index of all 
the Linked Terms and their associated Data Warehouses or 
databases appearing in the originating HTML document is 
dynamically created. Simultaneously, at step 755, 
infodex.pl, more specifically, the "frames" subroutine 
therein, creates two frames, FRAME 1 757 and FRAME 2 
756 in the browser's window. At 754, infodex.pl then sends 
the dynamically created index file to FRAME 1 757. 

Simultaneously, at 753, infodex.pl sends a static HTML 
file that explains how to use Infodex or other such useful 
information to FRAME 2 756. As indicated in block 758, the 
user can click on a link in the index (in FRAME 1 757) and 
send a request header to infodex.pl on the Proxy Server, 
which then sends a request header to the Document Server. 
The Document Server then sends the originating document, 
scrolled to the exact place of the Linked Term to the Proxy 
Server, or even directly to the browser, if that is desired. If 
the Document Server sends the file to infodex.pl on the 
Proxy Server, infodex.pl will forward the file to the browser 
and insert it in frame 2. At this point, the user can initiate the 
process illustrated in FIG. 5 by clicking on the highlighted 
Linked Term. 

FIG. 13 shows the browser's window as a result of 
operating the "Infodex" feature. As a result, left frame 803 
is shown after it is automatically loaded with a dynamically 
created HTML document 800 in HTML created by Java, 
JavaScript, C++, Visual Basic or ActiveX, or any combina- 
tion thereof, or any other relevant computer programming 
language. In document 800 the Linked Terms of the main 
document are presented in an expandable file tree index 805. 
By clicking on any one of the words, phrases, sentences or 
paragraphs with a symbol next to it, or on the "Expand" key 
801 at the bottom of left frame 803, a branch of the tree 
expands below the "parent" limb, revealing on the "child" 
limb the names of all the Data Warehouses or databases 
linked to the Linked Terms represented by the "parent" limb. 

At the same time, right frame 804 loads an HTML 
document 802 which explains how to use Infodex or other 
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such useful information. Both the right and the left frames 
are scrollable in either an up and down direction or in a side 
to side direction. 

FIG. 14 shows the browser's window including "parent" 
limb 900 and "child" limb 901 in left frame 803. The viewer 5 
may then click the "child" limb 901 in the left frame 803, 
which causes the viewer's browser to request and load the 
full multimedia document in the right frame 804 and scroll 
to the area of the document that contains the Linked Term to 
the selection in left frame 803. FIG. IS shows right frame 10 
804 loaded with a document 1000 including Linked Term 
1001 which is related to the "parent" whose "child" 901 the 
viewer clicked on. As shown in FIG. 15, the viewer will be 
able to see simultaneously, in left fame 803, an index of all 
the Linked Terms linked to a remote database or Data a 5 
Warehouse, and in the right frame 804, the full text of the 
document 1000 scrolled to the Linked Term and its sur- 
rounding text. If the viewer chooses, he or she may scroll 
from the beginning to the end of the entire document 1000 
in right frame 804. This document is the same document 20 
discussed above and illustrated in FIG. 6, and referenced as 
document 301. Accordingly, the viewer may click on the 
Linked Term in document 1000 and activate the same 
process discussed above and illustrated in FIGS. 4 to 10 
above. 25 

Thus, a cost-effective, highly efficient method and appa- 
ratus for automatically and continuously updating on 
demand an information source situated on a network 
resource has been described. Moreover, the method and 
apparatus has the additional advantages of causing the 30 
automatic updating of any document containing a linked 
Term with information (current or archived) to expand and 
add depth and context to any document. For example, it 
permits: 

Departments within corporations to update critical inter- 35 
nal documents, such as 401Kplan information, for their 
employees without labor intensive efforts; 

The information services department of corporations to 
distribute updated information on software used by 
employees by first describing the software in a text 40 
document, and then using the invention to link to any 
network resource which holds documents pertaining to 
the update. 

A corporation to have its sales force make personal sales 45 
calls, distribute print documents at the time of the sales 
call, and then update those distributed documents over 
a network, such as the Internet. 
Automatic updating of print documents handed out at 

seminars and conferences. 50 
Physicians to place descriptions of treatments for particu- 
lar patients suffering from illnesses in password pro- 
tected areas of a web site, and then have the informa- 
tion on the latest advances of drugs, or the legal status 
of FDA approval of drug updated regularly. 55 
While the present invention has been particularly 
described with respect to the illustrated embodiment, it will 
be appreciated that various alterations, modifications and 
adaptations may be made based on the present disclosure, 
and are intended to be within the scope of the present 60 
invention. While the invention has been described in con- 
nection with what is presently considered to be the most 
practical and preferred embodiments), it is to be understood 
that the present invention is not limited to the disclosed 
embodiments) but, on the contrary, is intended to cover 65 
various modifications and equivalent arrangements included 
within the scope of the appended claims. 
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For example, other gateway applications besides CGI, 
such as Sun Microsystem Incorporated's Java; Microsoft's 
Internet Server API (ISAPI) and Netscape's API (NSAPI), 
Java, Java applets, and Oracle's Web Request, may be used 
for transmission of data between the Proxy Server any other 
HTTP server. Also, immediately following the Linked 
Terms, graphic symbols for audio, video and text may be 
added to indicate to the reader the properties of the content 
to which the user gained access. Moreover, once the user 
accesses the database or Data Warehouse and views the 
information he or she sought, a window could appear which 
would allow the user to further refine the search by entering 
additional search terms and running another search in the 
database or Data Warehouse. Other embodiments would also 
include substituting std in and std out as a means to pass data 
through CGI to the Database HTTP Server instead of 
QUERY_STRING or PATH_JNFO environment variables, 
or using Java, Java applets or enhanced CGI as detailed 
above. 

What is claimed is: 

1. A method of dynamically augmenting the contents of at 
least one file of information on a first network resource, said 
file of information having at least one link, said method 
comprising the steps of: 

creating at least one request corresponding to said at least 
one link; 

coupling at least one query argument with said at least one 

request; 
providing a database; 

sending said at least one request and said at least one 

query argument to said database; 
causing said database to search for at least one record that 

satisfies said at least one query argument; 
providing a display for viewing by a user; 
subdividing said display into a plurality of frames; 
displaying said at least one record that satisfies said at 

least one query argument in at least a first frame of said 

plurality of frames; and 
providing a series of graphic symbols in at least a second 

frame of said plurality of frames for enabling the user 

to select one of a plurality of databases. 

2. The method of claim 1, further comprising the steps of: 
subdividing said display into a second plurality of frames 

which replace said plurality of frames; and 
providing a list of index terms in at least one frame of said 
second plurality of frames, said index terms for select- 
ing said at least one link in said at least one file of 
information on said first network resource. 

3. The method of claim 1, wherein said step of coupling 
further comprises coupling at least one authentication argu- 
ment to said at least one query argument and said at least one 
request. 

4. A method of dynamically augmenting the contents of at 
least one file of information on a first network resource, said 
file of information having at least one link, said method 
comprising the steps of: 

creating at least one request corresponding to said at least 
one link; 

coupling at least one authentication argument and at least 

one query argument with said at least one request; 
providing a database; 

sending said at least one request, said at least one query 
argument and said at least one authentication argument 
to said database; 
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causing said database to search for at least one record that 
satisfies said at least one query argument; 

providing a display for viewing by a user; 

subdividing said display into a plurality of frames; 

displaying said at least one record that satisfies said at 
least one query argument in at least a first frame of said 
plurality of frames; and 

providing a series of graphic symbols in at least a second 
frame of said plurality of frames for enabling the user 
to select one of a plurality of databases. 

5. The method of claim 4, further comprising the steps of 
providing: 

subdividing said display into a second plurality of frames 
which replace said plurality of frames; and 

a fist of index terms in at least one frame of said second 
plurality of frames, said index terms for selecting said 
at least one link in said at least one file of information 
on said first network resource. 

6. The method of claim 4, wherein said step of sending 
said at least one request is performed by a browser, further 
wherein said at least one request comprises a request header 
specifying: a) a purpose of the request; b) a network address 
of a second network resource to which said request header 
is applied; c) a file name for an application that is stored on 
said second network resource; and d) an argument that acts 
as a key in a table lookup, and corresponds to said at least 
one link. 

7. The method of claim 6, wherein said request header is 
sent to said second network resource via collaborative 
information systems transmission protocols used on a net- 
work. 

8. The method of claim 7, wherein said collaborative 
information systems transmission protocols used on a net- 
work include the combination of Hypertext Transfer Proto- 
col (H i IF) and Transmission Control/Internet Protocol 
(TCP/IP), including TCP/IP version 6.0. 

9. The, method of claim 4, wherein said coupling step 
includes the step of executing an application stored on a 
second network resource, said application causing the cou- 
pling of said at least one authentication argument and said at 
least one query argument with said at least one request. 

10. The method of claim 9, wherein said application that 
is executed on said second network resource uses as a 
gateway interface to the database a Common Gateway 
Interface (CGI) protocol interface. 

11. The method of claim 9, wherein said application that 
is executed on said second network resource uses as a 
gateway interface to the database a gateway interface 
selected from the group consisting of non-Common Gate- 
way Interface (CGI) programmic object oriented communi- 
cations protocols and non-CGI programmic server exten- 
sions. 

12. The method of claim 9, further comprising the steps 

of: 

determining whether a first authentication of a user is 
needed before said at least one request is sent to said 
second network resource; and 

performing said first authentication of the user if said first 
authentication of the user is required. 

13. The method of claim 9, wherein said at least one 
request comprises an argument that acts as a key in a table 
lookup and wherein said application executes on said second 
network resource and correlates in said table lookup said key 
with one of a plurality of expert predetermined optimum 
values, each said expert predetermined value comprising a 
network address for a database, a query argument, and an 
authentication argument. 
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14. The method of claim 13, wherein said key can be used 
to create a window display for viewing by a user, said 
window display presenting the user with a menu of choices 
for further areas of research pertaining to said key. 

5 15. The method of claim 14, wherein the user, by selecting 
one of said choices, causes said application that is executed 
on said second network resource to match a key, correspond- 
ing to said one of said choices in a table lookup, with a 
request header comprising: a) a purpose of the request; b) a 

io network address for a third network resource to which said 
request header is applied; c) a file name for an application 
that is stored on said third network resource; d) a query 
argument; and e) an authentication argument; and causes 
said application on said second network resource to send 

15 said request header to said third network resource. 

16. The method of claim 13, wherein said application that 
is executed on said second network resource matches one of 
said expert predetermined optimum values with said key in 
said table lookup selected from the group consisting of a 

20 hash table, an associative array and a linked list. 

17. The method of claim 9, wherein said application that 
is executed on said second network resource generates at 
least a second request header comprising: a) a purpose of the 
request; b) a network address for a third network resource to 

25 which said second request header is applied; c) a file name 
for an application that is stored on said third network 
resource; d) a query argument and e) an authentication 
argument. 

18. The method of claim 4, wherein said database authen- 
30 ticates a user; executes a search; and returns to a second 

network resource said at least one record that satisfies said 
at least one query argument. 

19. The method of claim 4, wherein an application that is 
executed on a second network resource modifies said at least 

35 one record that satisfies said at least one query argument. 

20. The method of claim 4, wherein an application that is 
executed on a second network resource causes a browser to 
subdivide said display into said plurality of frames. 

21. The method of claim 20, wherein said application 
40 executing on said second network resource causes said at 

least one record that satisfies said at least one query argu- 
ment to appear in the largest frame of said plurality of 
frames. 

22. The method of claim 20, wherein said application that 
45 is executed on said second network resource causes a 

plurality of buttons corresponding to a plurality of argu- 
ments for selecting a plurality of databases to appear in at 
least a second frame of said plurality of frames. 

23. The method of claim 22, wherein said plurality of 
so buttons comprise arguments that act as keys in a table 

lookup, with all said keys corresponding to a specific link in 
said at least one file of information on said first network 
resource. 

24. The method of claim 23, wherein the user, by clicking 
55 on one of said plurality of buttons, causes said application on 

said second network resource to match said key correspond- 
ing to said button in said table lookup with a third request 
header comprising: a) a purpose of the request; b) a network 
address for a third network resource to which said third 

60 request header is applied; c) a file name for an application 
that is stored on said third network resource; d) a query 
argument; and e) an authentication argument and causes said 
application on said second network resource to send said 
third request header to a third network resource. 

65 25. The method of claim 4, wherein an application that is 
executed on a second network resource causes an excerpt of 
text that includes said at least one link in said at least one file 
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of information to appear in at least a third frame of said 
plurality of frames. 

26. The method of claim 4, wherein an application that is 
executed on a second network resource causes information 
associated with the source of said at least one file of 
information to appear in at least a fourth frame of said 
plurality of frames. 

27. The method of claim 4, wherein the user may cause 
through a graphical device on said display to increase and 
decrease a viewing area of any one of said plurality of 
frames. 

28. The method of claim 4, wherein said at least one link 
is identified by icons selected from the group consisting of 
symbols representing text, audio, graphics and video infor- 
mation. 

29. A method of dynamically augmenting the contents of 
at least one file of information on a first network resource, 
said file of information having at least one link, said method 
comprising the steps of: 

providing a browser; 

providing a second network resource having at least one 
application; 

creating at least one request corresponding to said at least 
one link; 

sending said at least one request to said second network 
resource, wherein said second network resource, in 
response to receiving said at least one request, executes 
said at least one application and sends at least one 
embedded application to said browser; 

coupling at least one authentication argument and at least 
one query argument with said at least one request; 

providing a database; 

sending said at least one request, said at least one query 

argument and said at least one authentication argument 

to said database; 
causing said database to search for at least one record that 

satisfies said at least one query argument; 
providing a display for viewing by a user; 
subdividing said display into a plurality of frames; 
displaying said at least one record that satisfies said at 

least one query argument in at least a first frame of said 

plurality of frames; and 
providing a series of graphic symbols in at least a second 

frame of said plurality of frames for enabling the user 

to select one of a plurality of databases. 

30. The method of claim 29, further comprising the steps 

of: 

subdividing said display into a second plurality of frames 
which replace said plurality of frames; and 

providing a list of index terms in at least one frame of said 
second plurality of frames, said index terms for select- 
ing said at least one link in said at least one file of 
information on said first network resource. 

31. The method of claim 29, wherein said coupling step 
is executed on said browser by said at least one embedded 
application. 

32. The method of claim 29, further comprising the steps 

of: 

determining whether a first authentication of the user is 
needed before said at least one request is sent to said 
second network resource; and 

performing said first authentication of the user if said first 
authentication of the user is required. 

33. The method of claim 29, wherein said at least one 
request comprises an argument that acts as a key in a table 
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lookup and said at least one embedded application is 
executed on said browser and correlates in said table lookup 
said key with one of a plurality of expert predetermined 
optimum values, each said expert predetermined value com- 
5 prising a network address for a database, a query argument, 
and an authentication argument. 

34. The method of claim 33, wherein said key can be used 
to create a window display for viewing by the user, said 
window display presenting the user with a menu of choices 
for further areas of research pertaining to said key. 

35. The method of claim 34, wherein the user by selecting 
one of said choices causes said embedded application that is 
executed on said browser to match a key, corresponding to 
said one of said choices in a table lookup, with a request 
header comprising: a) a purpose of the request; b) a network 

15 address for a third network resource to which said request 
header is applied; c) a file name for an application that is 
stored on said third network resource; d) a query argument; 
and e) an authentication argument; and causes said embed- 
ded application that is executed on said browser to send said 

20 request header to said third network resource. 

36. The method of claim 33, wherein said at least one 
embedded application requests said one of a plurality of 
expert predetermined optimum values and a plurality of said 
keys, associated with said link, from said second network 

25 resource. 

37. The method of claim 33, wherein said at least one 
embedded application that is executed on said browser 
matches one of said expert predetermined optimum values 
with said key in said table lookup selected from the group 

30 consisting of a hash table, an associative array and a linked 
list. 

38. The method of claim 33, wherein said at least one 
embedded application that is executed on said browser 
generates at least a request header comprising: a) a purpose 

35 of the request; b) a network address for a third network 
resource to which said request header is applied; c) a file 
name for an application that is stored on said third network 
resource; d) a query argument and e) an authentication 
argument. 

40 39. The method of claim 29, wherein said database 
authenticates the user, executes a search and returns to said 
browser said at least one record that satisfies said at least one 
query argument. 

40. The method of claim 29, wherein an application that 
45 is executed on said second network resource modifies at 

least one record that satisfies said at least one query. 

41. The method of claim 29, wherein said at least one 
embedded application is executed on said browser and 
causes said browser to subdivide said display into said 

50 plurality of frames. 

42. The method of claim 41, wherein said at least one 
embedded application that is executed on said browser 
causes said at least one record that satisfies said at least one 
query argument to appear in the largest frame of said 

55 plurality of frames. 

43. The method of claim 41, wherein said at least one 
embedded application that is executed on said browser 
causes a plurality of buttons corresponding to a plurality of 
arguments for selecting a plurality of databases to appear in 

60 at least a second frame of said plurality of frames. 

44. The method of claim 43, wherein said plurality of 
buttons comprise arguments that act as keys in said table 
lookup with all said keys corresponding to a specific link in 
said at least one file of information on said first network 

65 resource. 

45. The method of claim 44, wherein the user, by clicking 
on one of said plurality of buttons, causes said at least one 
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embedded application that is executed on said browser to 
match said key corresponding to said button in said table 
lookup and causes said at least one embedded application to 
generate a second request header comprising: a) a purpose of 
the request; b) a network address for a third network 
resource to which said second request header is applied; c) 
a file name for an application that is stored on said third 
network resource; d) a query argument; and e) an authenti- 
cation argument. 

46. The method of claim 41, wherein said at least one 
embedded application that is executed on said browser 
causes an excerpt of text that includes said at least one link 
in said at least one file of information to appear in at least a 
third frame of said plurality of frames. 

47. The method of claim 41, wherein said at least one 
embedded application that is executed on said browser 
causes information associated with the source of said at least 
one file of information to appear in at least a fourth frame of 
said plurality of frames. 

48. The method of claim 29, wherein the user may cause, 
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ment to said at least one request and said at least one query 
argument, further wherein said second network resource 
sends said at least one request, said at least one query 
argument, and said at least one authentication argument to 
said database. 

54. The apparatus of claim 53, wherein said second 
network resource further comprises a memory, said memory 
storing a table look up having a plurality of expert prede- 
termined optimum values, each said expert predetermined 
optimum value having a network address for a database, a 
query argument, and an authentication argument, further 
wherein said at least one request comprises an argument that 
acts as a key in said table lookup, said key being correlated 
with one of said expert predetermined optimum values. 

55. The apparatus of claim 51, wherein said second 
network resource is a proxy server. 

56. An apparatus for dynamically augmenting the con- 
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decrease a viewing area of any one of said plurality of 
frames. 

49. The method of claim 29, wherein said at least one link 
is identified by icons selected from the group consisting of 
symbols representing text, audio, graphics and video infro- 
mation. 

50. The method of claim 29, wherein said step of pro- 
viding said second network resource comprises providing a 
second network resource having at least one applet, further 
wherein said second network resource, in response to receiv- 
ing said at least one request, sends said at least one applet to 
said browser. 

51. An apparatus for dynamically augmenting the con- 
tents of at least one file of information on a first network 
resource, said file of information having at least one link, 
said apparatus comprising: 

a browser having a display for viewing by a user; 
a second network resource coupled to said browser, 
wherein said browser sends at least one request corre- 
sponding to said at least one link to said second 
network resource, further wherein said second network 
resource couples at least one query argument with said 
at least one request, said second network resource 
further causing said browser to subdivide said display 
into a plurality of frames; and 
a database coupled to said second network resource, 
wherein said second network resource sends said at 
least one request and said at least one query argument 
to said database, said database comprising a search 
engine for searching for at least one record in said 
database that satisfies said at least one query argument; 
wherein said at least one record that satisfies said at 
least one query argument is displayed in at least a 
first frame of said plurality of frames and a series of 
graphic symbols are displayed in at least a second 
frame of said plurality of frames for enabling the 
user to select one of a plurality of databases. 

52. The apparatus of claim 51, wherein said second 
network resource causes said browser to subdivide said 
display into a second plurality of frames to replace said 60 
plurality of frames and further wherein a list of index terms 
are displayed in at least one frame of said second plurality 

of frames, said index terms for selecting said at least one link 
in said at least one file of information on said first network 
resource. 65 

53. The apparatus of claim 51, wherein said second 
network resource couples at least one authentication argu- 
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resource, said file of information having at least one link, 
said apparatus comprising: 

a browser having a display for viewing by a user; 

a second network resource coupled to said browser, 
wherein said browser sends at least one request corre- 
sponding to said at least one link to said second 
network resource, further wherein said second network 
resource, in response to said at least one request, sends 
at least one embedded application to said browser; 
wherein said at least one embedded application is 
executed on said browser and couples at least one query 
argument with said at least one request, further wherein 
said at least one embedded application causes said 
browser to subdivide said display into a plurality of 
frames; and 

a database coupled to said browser, wherein said browser 
sends said at least one request and said at least one 
query argument to said database, said database com- 
prising a search engine for searching for at least one 
record in said database that satisfies said at least one 
query argument; 

wherein said at least one record that satisfies said at 
least one query argument is displayed in at least a 
first frame of said plurality of frames and a series of 
graphic symbols are displayed in at least a second 
frame of said plurality of frames for enabling the 
user to select one of a plurality of databases. 

57. The apparatus of claim 56, wherein said second 
network resource causes said browser to subdivide said 
display into a second plurality of frames to replace said 
plurality of frames and further wherein a list of index terms 
are displayed in at least one frame of said second plurality 
of frames, said index terms for selecting said at least one link 
in said at least one file of information on said first network 
resource. 

58. The apparatus of claim 56, wherein said at least one 
embedded application is executed on said browser and 
couples at least one authentication argument to said at least 
one request and said at least one query argument, further 
wherein said browser sends said at least one authentication 
argument, said at least one request and said at least one 
query argument to said database. 
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59. The apparatus of claim 56, wherein said at least one 
embedded application is executed on said browser and 
causes said browser to send said at least one request and said 
at least one query argument to said database. 

60. The apparatus of claim 58, wherein said second 
network resource further comprises a memory, said memory 
storing a table look up having a plurality of expert prede- 
termined optimum values, each said expert predetermined 
optimum value having a network address for a database, a 
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query argument, and an authentication argument, further 
wherein said at least one request comprises an argument that 
acts as a key in said table lookup, said key being correlated 
with one of said expert predetermined optimum values. 

61. The apparatus of claim 56, wherein said at least one 
embedded application comprises at least one applet. 
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